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EXECUTIVE SUMMARY 



Overview 

This report describes the results of a project sponsored by the Department of Education 
Small Business Innovation Research Program. In March, 1984, Creare completed the first 
phase of this project, the results of which were iwesented in a final report entitled, "Feasibility 
Study of An Educational Data Acquisition System for Use in Secondary Schools." 

We began Phase Two in July, 1985, under contract number 400-84-0006. We 
designed, prototyped, and then evaluated an Educational Data Acquisition System (EDAS) in a 
local high school The objectives of the Phase Two project were: 

1 . define the range of educational needs for EDAS, 

2. establish technical descriptions of the educational data acquisition system, and 

3* evaluate student and teacher response to automated laboratory science. 

All of these objectives have been achieved in the Phase Two project as discussed, in 
turn, below. 

Activities 

In Phase One and at tiie beginning of the Phase Two project we reviewed the existing 
data acquisition products in use in schools and the acceptance of these devices by the schools. 
Their usage was very low and by examination of the products themselves, talking to teachere 
who were and were not familiar with data acquisition and surveying the literature on computer- 
aided instruction in the science classroom, we developed a definition of schools' requirements. 
These requirements we^ reviewed by our educational consultants and were then used in our 
design of the EDAS hardware and software and in our evaluation criteria for the EDAS. 

During the first four months of Phase Two we pt;rformed the technical design of the 
EDAS hardware and software components. The design incorporated the concepts that Creare 
uses for its industrial data acquisition systems, results from our market research on 
micrccomputcr-bascd data acquisition systems, and the educational requirements from our 
surveys. We spent the next six months developing the prototype system, followed by a trial 
period of two months of preliminary usage and evaluation in the school. We then spent tiiree 
additional months updating the prototype to incorporate improvements suggested by students, 
teachers, and evaluator feedback. The prototype was then placed in the school for a year of 
testing. The effectiveness of the design implemented in the prototype EDAS led to a successml 
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evalu?,tion and favorable response of students and teachers across a wide variety of subject 
areas and grade levels. 

The EDAS evaluation lasted for an entire academic year. A research assistant in 
permanent residence at the high school obtained classroom observational data, information 
about student and teacher attitudes, and the results of students' written woric for all subject 
areas and grade levels throughout the year. The impact of the EDAS and the benefits and 
pitfalls of automated data collection in the science classroom learned from the evaluation will 
have major significance in future development of data acquisition products for the classroom. 

Organization of the Report 

The following three chapters present the results of the Phase Two project These 
chapters are organized as a series of three pi^fessional articles about the EDAS. Each chapter 
has been or is being submitJcd to refereed journals for publication. While this plan of 
pubUcaticm ensures wide communication of our results it does result in some unavoidable, but 
minimal, redundancies across the three chapters for the reader of this report Chapter I 
presents «ai overview of botfi the technical and evalu ive pwtions of the project, emphasizing 
the benefits of the EDAS. The analysis of the data in Chapter I is based on two case studies 
from the 19 experiments that were run. This type of analysis can sometimes illuminate 
information that is not revealed by the overall analyses of Ch^ter m, but can miss information 
that the methods of Chapter HI can detect Chapter n describes the details about how the 
educational evaluation was set up and operated. It is designed to serve both as an aix:hival 
record of what we did and as a manual for others who wish to apply similar evaluation 
methods to other problem areas. Chapter HI presents the detailed results of the evaluation. We 
en^loyed innovative methods in Chapter EI to examine simultaneously the results of 19 
different experiments. 

Conclusion 

We have met all the requirements of the project We developed a prototype EDAS for 
use in secondary science classes and had it rigorously evaluated by educational reseaix:h^. 
The results show that computerized data collection in the classroom will foster excellence in the 
sciences and allow students to achieve their full potential. 
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CHAPTER I: OVERVIEW OF THE PROJECT 



A Collaborative Project 
to Develop Computer Bassd Instruction 
in Science Laboratory Courses 

by G. Christian Jemstedt 
and 

Paul H. Rothe 



This ch^ter describes the results of an intensive three year program to bring the 
sophistication of industrial technology to the aid of science teaching and learning. With 
support from the Department of Education a team of experts was assembled from Creare, Inc., 
an advanced engineering consulting company specializing in computer software and systems, 
from the Psychology Department at Dartmouth College, a department with specialists in 
learning, curriculum design, and evaluation, and from Hanover High School, an excellent 
regional school with a six person science department that offers the basic scientific disciplines 
for a range of student ability levels. 

The team had two goals. The first goal was to take the best ideas for improving science 
education mm computer based tools and design ana create dn Educational Data Acquisition 
System (EDAS). The second goal was to evaluate the effectiveness of the EDAS m actual class 
use m a school system, asking (a) will the system work as designed, (b) can and will teachers 
use It m class, and (c) will the system lead to desirable changes in students' attitudes and 
performance? 

The Design Criteria for the Operation of the EDAS 

In laboratory science computers are a critical element of modem research. Computers 
play an essential role for laboratory scientists, engineers, physicians, office workers, 
executives, factory workers, artists; the list is nearly endless. The core of the EDAS was 
d<»igned around a personal computer to address the broad need for true computer literacy in 
science while improving the educational process as well. Although it was primarily designed 
for and was tested in secondary school classrooms, the EDAS can theoretically be used with 
students of nearly any age. 
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The overall goal in developing the specific apparatus was to provide flexible tools for 
the teachers to use in developing curriculum rather than rigidly fixed or required laboratory 
instoiments and procedures. At Creare specialists in software, hardwaze, and documentation 
designed the computer system. To assist in answering questions about what works in the 
classroom and what teachers will accept and use, the design process began with an information 
review that identified the existing computer data acquisition devices in use in the schools and 
their acceptance by teachers. A survey in 1982 of 1200 schools showed that only 12% were 
using computers in science classes (Harvard, 1982). Only 2% of that 12% ( e.g., 3 schools, 
or 0.2% overall) actually used a laboratory interface in their curriculum. Reviews of such 
software and hardware (Ahl, 1983), product literature, conversations with teachers who use 
the products, and hands-on evaluation of some of the products indicated that there were good 
reasons why use of such interfaces was low. The problems uncovered includwi the relatively 
passive role of the students in using some of the instruments, the fixed, limited nature of what 
can be done with the instruments, the absence of productive software to employ effectively the 
instruments, and the very limited, almost game-like quality of much of the ^paratus. 

Based on the research into computers in science, the EDAS design criteria differed ftom 
those of most current school instrumentation products in that the criteria specified a general 
purpose, industrial quality, data acquisition and reduction system. The school environment is 
probably harder on instruments than is the research laboratory. We decided that students and 
teachers need the same quality and ease of use that is available in industry; they need equipment 
and curriculum that emulates industrial scientific research. The EDAS should unobtrusively 
guide students and teachers into effective scientific activities. Toward that end, a set of eight 
features was designed for the EDAS: 

• fast, 

• accurate, 

• patient, 
dedicated, 

• efficient, 

• general purpose, 
modular, 

• comprehensive, 
uniform in operation, 

• orderly, 
flexible, 

• simple to use, and 
easy to use. 

The first four features of the EDAS differentiate it from equipment used by students in a 
manual, traditional experiment. The speed of the EDAS should allow students to exploit 
sensors and investigate processes that they could never consider in the past, while its patience 
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allows It to perform large numbers of repetitive operations with quick, efficient instructions 
given by the student By being dedicated to a measurement task the EDAS can collect data 
from long-term experiments, such as weather sampling over the period of days, weeks, or 
longer. 

The single most important feature of the EDAS is that it is general purpose in its 
application. Teachers and students vary widely in their abilities and familiarities with 
computing and science. The teacher and student should not be constrained to pre-programmed 
or canned laboratory work. They must have the power of a tool that ser\'es tiieir own, 
individual needs, and they should be encouraged to explore and define those needs. The 
modular nature of the EDAS allows it to be configured easily for different science courses. In 
a moment a new sensor module can be attached, changing the EDAS from a biological to a 
chemical instrument Yet, the software interface and general operating procedures remain 
constai^scrving to unify expcnments in various disciplines. We also envisioned moving the 
EDAS from one classroom to another during the class change period, which would allow it to 
be used more or less cwitinuously and thus significantly reduce its cost per classroom. 

l"he EDAS should be comprehensive in that it should contain all of the elements 
necessary to conduct en experiment, from data acquisition to final evaluation. The user must 
be freed to think about the data and their meaning, not the logistics of idiosyncratic, often 
obsolete ^paratus. Employing an orderly, but open menu approach guides the user in 
developing an orderly process of experimentation. Since the menu provides options, not 
requirements, teachers may tailor activities to the special needs, abilities, and interests of 
students. Analytically minded students might spend a greater proportion of their time with data 
handling functions, while mechanically minded students might spend more time in the actual 
conducting of the experiment 

The tase of use should make exploration with the EDAS simple, even fun. We wanted 
a system that would disappear into the background as the student thought creatively and openly 
about the process that he or she was investigating. We hoped for lots of "what if?' questions 
from the users, who would then proceed to answer the questions experimentally. As the 
system is shared across courses, students should build a common unc'erstanding of scientific 
method and be encouraged to biing what they have learned in other science courses into each 
new one. 

The Design Criteria for the Educational Impact of the EDAS 

We identified five critical features of effecti 'e educational environments that must be 
present m the science classroom and that must be supported by the use of the EDAS. As they 
form the educational foundation on which the EDAS development rested, they will be reviewed 
in some depth. 
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Interactive 

an active learner 

a responsive environment 

an orientation towards retrieval processes 

creation of a base of experience on which to build later 
learning 

Accessible 

attention getting 

challenging 

flexible 

cognitively structured 

efficient 
Collaborative 
Personalized 

use of familiar curricula 

manageable unit format 

flexible pacing 

mastery criteria 
Generalizable 

The EDAS is an interactive tool Students who are more active in their engagement 
with their subject matter and their classroom peers learn more (Webb, 1983). Furthcuaore we 
know that the study activities of such students serve as a catalyst to increase the effects of their 
motivation and abUity (Jemstedt & Chow, 1980). Due to its speed, efficiency, and patience, 
the computer in the EDAS is always waiting for student action. This implicit demand, 
characteristic of the computer, is not seen by students as pressure so much as positive incentive 
to engage, as evidenced by the computer's positive effect on classroom attitudes (Kulik. 
Bangert, & Williams, 1983). 

Generally, as student work is monitored and feedback is provided, learning improves 
(Berhner & Rosenshins, 1977). When students work in groups, ^-^wever, teachers cannot 
provide effective individual monitoring of student activities. The ability to monitor 
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performance and respond imnvdiately, directly, and in a manner that is focused on the 
important questions is a necessity in science courses where long sequences of steps may be 
taken before an answer, the feedback to the student, is achieved The EDAS will thus serve a 
responsive role in the classroom by directly requesting and reactmg to students' actions, such 
as their connecting of sensors and entering of information, data, or commands. 

One of the most basic needs in the classroom is for an orientation on the retrieval (rather 
than the storage) mental processes of the learners. Learners must rehearse what they have 
learned; it is the crucial learning process, and we have known about it for many years (Spitzer, 
1939). As described earlier, it is not what learners see or hear, but what they do that most 
determines what they Icam. This retrieval fccus is aided by learning materials and activities 
that are structured, provide imagery, and encourage elaboration* 

Let us consider structure first The science curriculum can be described in terms of 
lower and higher order rules. The use of a few well chosen hi^er order rules can drastically 
reduce the number of lower order rules diat a student must learn. Moreover, the students will 
pcrtbrm just as well on problems that require die use: of the lower order rules diat were 
eliminated (Scandura, 1977). The modular and general purpose design of die EDAS is based 
not only on economic foundaticms, but also on tte desire to present the concepts of laboratory 
science as a relatively few, broadly aR)licable principles of data collection, and analysis, and 
presentation. The fact that students will encounter a common set of high level tools in each 
laboratory exercise ensures mastery of those major principles. 

The retrieval process is also aided by die use of imagery. Much of current educational 
software, even for high school science, has its roots more in the imagery of comic books than 
of scientific illustration. Scientific graphic image presentation, of the form present in the 
EDAS, can facilitate learning and improve retention (Wittrock & Lumsdaine, 1977). 

The asking of questions that require deep intellectual processing of information can be 
done almost ideally ii; the science laboratory (Ciaik & Lockhart, 1972). Traditional science 
exercises are often heavy on manual woric while light un intellectual activity. The relative 
efficiency with which hypotheses can be tested with the EDAS can minimize the drudging 
routine activities and hence encourage a greater degree of thought and questioning, both on the 
part of teachers of their students and by students themselves. The result of such activities is 
increased retention (Horton & Mills, 1984). 

More than any other area of the curriculum, science requires direct student experience. 
There is a vast gulf between the actions of describing a scientific principle and tiiose of 
corrccfly using that principle (Simon, 1980). If students are to think like scientists tiiey must 
begin to learn how to act like scientists. The experienced individual views events and ideas in 
ways that are fimdamentally different from those of Jie inexperienced individual (Reed & 
Johnson, 1977). As students with the EDAS increase the proportion of their time Aat is spent 
in intellectual woric rather than clerical activities, ihey will acquire more extensive experience in 
die fundrmentals of creative science. 

Though they may be well designed, learning materials are of minimal effectiveness if 
they are not perceived as accessible to student and to teacher. This accessibility can be 
improved through using science apparatus that is attention getting, challenging, flexible, 
cognitively structured, and efficient 
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Conq)uters in the classroom can produce a higher level ol student attention (Jemstedt, 

1983) . Furthermore, increased attentiveness is significantly correlated with achievement 
Because the EDAS iKovides a familiar lab<»atoiy interface for a wide variety of experiences, it 
will avoid the "drowning ouf effect that accompanies the use of continually novel materials 
(Hontkowski & Calfee, 1979). 

Though science is a lively intellectual area, with changes in knowledge occuiring daily, 
science laboratories often deal with topics that have been resolved for decades, if not for 
centuries. The excitement of open experimental exploration with the EDAS should challenge 
students to search for information and induce mac rehearsal of what is happening in the 
laboratory (Smith, Johnson, & Johnson, 1981). The flexible, open-ended nature of the "what 
if' questions that are possible with the EDAS, compared to the specific question answering that 
is typical of traditiaial science laboratay experiments, should suppcMt loig term retention*of 
the material. 

Cggnitivc StniCturg is an additional design criterion that must be considered carefiilly, 
since increased, ^ropiiate cognitive effort increases learning and retenticMi (Horton & Mills, 

1984) . Because the user must select each step of the experimental process from EDAS menus, 
Ae EDAS cannot be operated without knowing what one is trying to do; Aus Ae EDAS menus 
are designed to require intellectual effort 

We have discussed accessibility to Ae student, and turn now to accessibility to Ae 
teacher. The activities of a teacher in a subject matter such as science are quite complex and 
resemble Ae skills needed to run a small business (Duke, 1979). But rarclv are teachers 
supported wiA the tools that are typically available in such a business. The efficiencv of Ae 
computer in handling many different students and many different experiments while producing 
caiceptually unif<xm reports will help teachers achieve some of the productivity gains Aat 
computers arc bringing to Ae business worid. The increased teacher productivity can produce 
oAer desired effects. In place of Ae time normally spent for laboratory set up, take down, and 
o Aer such manual work the teacher's job wiU be to fi^e questions whose answers will 
illustrate Ae critical principles of the subject matter. The teacher can focus <mi Ae intellectual 
matters more Aan Ae logistical ones. 

The superior learning Aat occurs wiA collaborative efforts among students has been 
known for some time and studied by many investigators (Johnson & Johnson, 1975; Johnson 
Mamyama, Johnson, Nelson, & Skon, 1981; Sharan, 1980). Within a class or student lab 
group Ae various tasks involved in using the EDAS can be apportioned according to student 
skill level and interests. One of the strengths of collaborative learning is that each member of a 
group can feel that he or she is a valued contributor to Ae group process. 

Science education, to be effective, must be personalized for boA Ae individual 
teacher and the individual student A major problem wiA federally supported curriculum 
materials for science education in Ae past has been Ae resistance of teachers to fixed, new 
curricula (Welch, 1979). This problem can be ameliorated wiA Ae EDAS by allowing Ae 
teacher to combine his or her own mix of self-chosen, familiar materials wiA a set of standard 
EDAS operating procedures. 
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For the student, personalization focuses largely on dividing material into manageable 
]mit& that can be mastered at a rate appropriate to the level and background of the student 
(Ryan, 1974). The goal of small units that can be completed widiin one class period can be 
achieved with the speed of the EDAS (Semb, 1974). When units are small, the pacing of a 
course becomes less troublesome and more fxcxible. Some students may complete two units 
while others complete only one. 

The use of a master y criterion is the most common characteristic of personalized 
courses of instruction. Requiring students to know funaa*Ttental skills well, before moving on 
die men advanced skills, is of critical impcMtancc in science educaticm. The research is clear 
that mastery produces better outcomes (Hursh, 1976). With the EDAS students can easily aiid 
quickly repeat experiments, with new parameters, until thqr master the underlying concepts. 

Effective retention ci what has been learned in the science laboratc^ depends in part on 
the degkiee to which cues in the new environment generalize because they are similar to those 
in the original learning cnvin)nnient(moscovitch&Craik, 197^^ For this reason, if students 
are to take v^ Aqr leam in tte school sdence course with them in 1^ 
w<xkenviionmrats,dK)se environments must be similar. Con^ioers are an essential part of 
modem industrial laboratcmes, and, voy recently, of university science laboratories. The 
d^rce to which students enccnmter them in realistic ways in high school will detomine the 
degree to which their high school course is generalizable. In other areas it is clear, and may be 
so in the sciences as well, diat increased exposure to instructional con^uters can inq)rove 
attitudes toward the use of computers as tools and toward the courses in which the computers 
are employed (KvJik, et al, 1983). 

The Basic EDAS 

The structure of the EDAS is illustrated in Rgure LI. The host microcomputer is a 
standard IBM PC with monitor, disk drive, serial interface, and printer. The machine has far 
mwe power than other classroom computers and, with the advent of IBM clones, is quite 
affordable. TTie electronic interface (DAQ) is a Burr-Brown 3002 Data Acquisition System 
containing an analog to digital converter and its own microprocessor and memory. Signal 
processors within tfie DAQ ccmditicm the data received from various sensors. The sensors 
allow the DAQ to san^k essentiaUy any physical process, including temperature, pressure, 
force, moticm, pH, ccmductivit/, s(mnd, light, and electricity. 

The key to the effectiveness of the EDAS for the user is the software that opera 
systeniL Much of the {Hoject effort focused on developing software tfiat was truely flexible and 
easytouse. Menus offer tfie user a full ranee of options: fa) prepare for an gxperimenL 
including changing the e>q)erimental scbsp by adding or deleting sensors and checking the 
status of sensors; (b) nm an experiment^ including setting the operating mode to storage of data 
or monitoring and display of data, setting the sam* iling rate, setting the display format to 
gr^hical or tabular, and collecting data; (c) redu::e and display data, including selecting data 
fUes and displaying their contents; (d) manage the data dkkg. including cataloging, deleting, 
and copymg files; (e) change the cnnfipiration n f the system, including specifying the 
computer and peripherals present and the types of sensors: and (f) receive help. No knowledge 
of programming or computer operation is required other than the ability to turn the machine on. 
The student scientist at the control of the machine thinks about the experiment, not the 
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computcn The soft\ vaie and system operation are further designed so that teams of students 
can use the machux, mV\ each pcrfwming a different p.'rt of the experimental procedure. 

A foil txplandtucy manual was written for the EDAS as part of its development We 
forsee the eventual creation of other suppcHting materials including a teacher's guide to the 
pedagogical aspects of the EDAS, individual guides for a wide variety of laboratory 
experiments^ and r ncwslettc r for the sharing of other ide**-; about the effective use of tiic EDAS 
in the classroom* 
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SOFTWARE 

prepare for experiment 
run experiment 
reduce and display data 
manage disk 

change system configuration 
obtain help 




DATA ACQUISITION SYSTEM 

i i i 



signal I 


1 signal { 


1 processor 1 


{^processor | 



1 sensor i 



1 sensor m 



Figure 1. 1. A schematic representation of the design of the ED AS. 
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Evaluating the Effectiveness of the EDAS 

We tested the EDAS's effectiveness by spending a full academic year conducting true 
experimental research on its actual use in the school. This year of evaluation was preceded by 
three months of pilot studies, conducted during the previous academic year, in which we tested 
the EDAS and all materials that we planned to employ in the full evaluation. During the pilot 
study we developed a set of desired modifications to the EDAS that were based on student and 
teacher feedback. Three months were then spent in refining the hardware and software to 
reflect these suggested improvements. During this three month period all initial data were 
analyzed and the evaluation procedures fine tuned. 

There are two different styles in which an instrument like the EDAS can be employed in 
class, and we evaluated both. Fot some laboratories the instructor presented a demonstration 
of a laboratory principle for the entire class. The role of the students was to observe the teacher 
and answer questions during the denjonstration. Fot other laboratories students conducted 
their own experiments under the teacher's guidance. Two stations were available in class, an 
EDAS station and a preparation station, so that the lab groups of 2 to 4 students could prepare 
for their experiment at one station and then move to the EDAS station to conduct the actual 
experimental woric In this way a fiill section of students could share the EDAS, with each 
performing all actual work on the computer. 

More Aan four hundred students enrolled in six laboratory science course participated 
m the study. There were courses in general science, biology, chemistry, and physics. Detailed 
data were collected about the students' abilities and academic history from their transcripts. 
The evaluation of the EDAS was conducted in a series of 19 true experiments. In each 
experimen t the EDAS was compared with the traditional laboratory ^paratus used in the 
course. Since each of the six sciences courses met in several different sections, we were able 
to have the same teacher in each course jMcsent the laboratory with the EDAS for one or more 
sections and widiout flie EDAS for one or more sections. To ensure that the effects we 
observed were not simply due to the novelty of the EDAS, we varied which sections received 
which set of instruments so that all students and all teachers were familiar with all methods and 
expenenced all forms of instruction. When the EDAS was used in a section, all students in the 
section learned with the computer. Comparisons between the EDAS and the traditional 
apparatus were, therefore, made between different sections of the course. Appropriate steps 
were taken during the data analyses to test and control for effects of differing student abilities in 
the various sections of the course. 

A full time research assistant and field .imager was placed in the high school 
throughout the evaluation period This person prepared all evaluation materials, observed 
student behavior in each class, entered and analyzed data, and performed all the ancillary 
activities associated with evaluation. In this way the teachers could proceed as they normaUy 
would have done in the courses, and disruption due O the evaluation was minimized. 

We obtained information about the effectiveness of the EDAS from four sources, 
including semester questionnaires, daily questionnaires, class observations, and couree grades: 

( 1 ) To collect information about students' long-term motivations and 
attitudes, we used semester questionnaires. At the beginning and end of each semester 
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students filled out a form that explored their attitudes, study habits, learning style, personality, 
school activities, and feelings about their courses and their own academic performance. 

(2) To examine short-term attitudes, we had students fill out brief forms 
each day that a laboratory was offered, just before and just after class. These daily forms 
assessed students' enjoyment of the class, interest in the subject matter, the difficulty of the 
class, and tiieir enjoyment of and comfort with computers. In addition, the form given before 
class asked about Iheir motivation to leam and do well in the upcoming laboratory session, 
while the form administered after class added question- about the amount of knowledge they 
had leamal, their confidence in what they were leami g, the effectiveness of the procedures 
that th^ were using, the degree to which they worked on their own, and the degree to which 
they felt rushed while the class was in session. Teachers filled out a similar form with their 
class. 

(3) To understand the EDAS impact on the actual activities of students and 
teachers in class, we observed student and teacher behavior during each laboratory session. 
During the labaatories &e project field manager systematically collected continuous 
observations about Ac bchaviw of the students and the teacher. Every 60 seconds the observer 
recorded on a coded sheet which of three categwies of behavior the teacher was engaged in and 
the proportion of students in the class that were engaged in each of six categories of student 
behaviOT. The teacher's behavior was categorized as not attending to the student providing 
individual attenticm to stunts, or talking to groups of students. The students were examined 
for the degree to which they showed on-task or off-task behavior, that is, the degree to which 
they were actively and directly engaged in the laboratory assignment The six types of student 
behavior were: off-task, group oriented on-task, individual engagement with the teacher, 
engagement with their group and the teacher, attending to the apparatus, and writing down data 
or notes. 

(4) The overall impact of the EDAS on cognitive performance was 
monitored by analyzing the nwmal examinations and laboratory reports assigned by the 
teacher. For each labcratory sessicm, data were collected about the student's actual academic 
performance. Tests scores were divided into scores for questions that dealt with laboratory 
cwicepts and those that dealt with other aspects of the course. Examination questions were 
separated according to whether they were directed towards fact recall or ^plication skills. 

Extensive multivariate analyses of all data were conducted. With 19 different 
experiments arid the continuous attitude data, we obtained an enormous amount of information, 
some 200,000 numbers. The results presented here provide but a sample of the overall results, 
more technical descriptions of the impact of the EDAS will be made available elsewhere 
(Chapters II and ffl). Fot the sake of clarity and coherence, the data from one course is 
presented here. This course, an intermediate level chemistry course, is representative of the 
other courses we examined. 

The Impact of the EDAS 

Thi impact of the EDAS was manifested in three domains: the beha vior during the 
laboratory of the students and the teacher, the students' attitudes about the laboratory, and the 
students' performance on test instruments after the laboratory was finished. There were 
differenct^ between students who learned with the EDAS and those who did not in all three 
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Figure 1.2. The classroom behaviors of the teacher and students during typical EDAS and 
traditional laboratory sessions at two points in the academic year. 
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Significant differences in students' performances and students' attitudes about 
themselves, the course, and computers also emerged as a function of whether or not students 
used the EDAS. The attitudinal differences changed across the academic year. The attitudes 
and performance data are presented in Figure 1.3. 

Early in the year students who used the EDAS reported less confidence in their abiUties 
dunng the laboratory and more frequent feelings of not knowing what they were doing. 
Despite this uncertainty, the students in the EDAS sections were more positive in their attitudes 
towards computers at the end of each class than were their peers in tradfdonal sections. The 
actual performance of the students on the examination and laboratory reports produced 
outcomes opposite to the attitude effects. Students who learned with the EDAS wrote better lab 
reports of their wo± and performed better on examination questions about the laboratory 
exercise. Thus it seems that the EDAS students were at once performiu, better and challenged 
to perform still better. ^ 

Later in the year, the students using the EDAS in their section were equally confident 
and sure of what Aey were doing. Furthermore, they feU that tlie laboratory work was actually 
easier with the HDAS and reported that they felt less rushed than the sfidents in the traditional 
swtions reported This did not lead to enhanced enjoyment of clas' lowever. Students using 
flie traditional apparatus reported more enjoyment of class than did i dents using the EDAS 
It IS not clear how or whether this difference influences student behavior, it deserves further * 
study. For the EDAS students the examination scores were consistently as good as or better 
than those for the students using traditional instruments. 

Throughout the year a particular effect emerged suggesting that the improved 
performance on examinations for the EDAS students resulted from an increased ability to apply 
the laboratory pnnciples to new problems. Rather than simply learning more facts, the EDAS 
students were learning how to bring what they had learned to new situations 
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Figure 1.3. Student attitude and pcrfonnance comparisons for the EDAS and traditional 
instruments at two points during the academic year. 
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When the EDAS was used for demonstrations rather than hands-on laboratory 
^'^EJT"??^'.*? ^^f^} different In general, students felt more rushed when the teacher 
usai the EDAS and felt that the use of the EDAS was less effective than the use of the 
traditional mstruments. It is likely that this represents the teacher's awkwardness and 
nervousness with using the EDAS. Field observations indicated the presence of greater 
nervousness on the teacher's part when the computer was employed, presumably because the 
teacher lost the benefits of years of experience that he or she had with the traditional 
insmiments. It is important to note, however, that examination performance did not correlate 
^1. .t'^^Slf'*"^'- «>"ti""«i to score as weU or better on examinations for 
which they had been prepared with the EDAS. Ai noted with the hands-on use of the EDAS, 
newSti^s perfortnance ip.volved an abUity to apply the laboratory principles to 

The Benefits and Pitfalls of Computers in the .chool Science Laboratory 

»c ^ .cZJi^n^ f ^iP^i"* witfr three questions, asking (a) whether the EDAS would operate 
as dKigned, (b) whether it would be used well by the teachers, and (c) whether it would 
mfluence the st»dente attitudes or performance. The field evaluation confirmed that the 

AiTftlf ^ P^"^^^ tool that can improve science learning in crucial ways, but reminded us 
tnat It IS not a panacea. 

♦u ^® ^^^i ^ ^ scientific instrument system, performed outstandingly well. It met all 
the operational design requirements we set for it The engineering of tools is a relatively clearly 
understood process, andTt proceeded weU for us. The combination of engin^re, ^ ^ 
J!!? cc ^"l*"'^ teachers produced an exciting and satisfying design and development 
process, me team approach can provide exceptional tools to serve educational needs. 
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Rg nIA Students and teacher engaged in data collection with the EDAS. Note the degree of 
on-task attention from each. The EDAS is in the top right and the materials being examined in 
the center left of the photograph. The teacher is in the right foreground. 
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Wc found that after its development, tfie actual introduction of an innovative, 
technological tool is a complex process with a great number of variables influencing its use in 
the school We encountered no absolute limiis to what teachers can achieve, but we exposed 
problems that must continuously be dealt with, including the adjustment to a new and different 
tool with its concomitant change? in attitudes. Great tools need skilled yscrs to acnieve their 
full potenuaL The teachers and students worked cnthusiasdcally to learn how to use the 
EDAS, yet were sometimes discourag«5d and confused early in ther learning. 

Tbt skills needed to help the teachers with their adjustment to nev/ technology are those 
of the professional tramer. For the future, traimng materials must be developed with the help 
of teachers and students who are funilliar with the use of coo^uter-based laboratory 
instnimmts. materials shouW riot be canned labonitoiy experiments; rather, they should 
be exercises that let teacher and student come to undentarid the potential for this new t^ 
howto<»enueitmosteerectiveW. The tnuning could be brief and directed towards pnducine 
a tevd of comfort in dealing with ♦he convttter. It laight involve a two day woikshco for 
teachm, videoti^ sessions with nutter teachers, or simply the presence of a moti^^ H 
colleague m the school who has used the system. We noted in the test high school that teachers 

WW attracted to usmg the EDAS as they heard firom their ooUeagucs and f^ 
others' success with it 

"nie use of the computer by teachen for class demonstrations requires furthers^ 
One of tte authors has used a labootoiy con^ter to conduct claos demonstrations for large 
groups of students for many yean. It took the first few years for him to learn how to make 
most effective use of the computer in that mode of operation. Then the computer supported 
class meetinM tha» v-re far more effective in impact on both student attitude and perfranance 
than those with the traditional instruments. Since the abi% to ask the right question takes 
extended experience; one must be prepared 'or a learning curve wi A new technoloev and 
teaching methods. 

In many arenas we found that while students are good at reporting most aspects of their 
behavior they are not paiticulariy ^ at estimating what they know and whether fliey arc 
proceeding along the cooect learmng path. Low correlations between student attitudes and 
actoal pcrfonnancc iqjpttrr not to be umquc to the EDAS; ther w comm^ 
educational research. The teacher can help students with this matter by assisting students in 
imdcrstanding and predicting the direction in which their classroom explorations might take 
them. 

The presence of the EDAS also stimulated a great deal of behavior directed towards 
curriculum expkxatioa and inqwovement TTie flexibility and versatility of the EDAS enabled 
teachers to develop new laboratoiy exercises that were simply impossible with traditional 
msbuments. The ability to repeat experiments rapidly and with different parameters allowed 
students to opoate in a iiiore creative and exploratory styte in c^^ Attention shifted from the 
mecharm of the UOwatoiy procedures to the princq)les behind tte Thisshiftis 
reflected in the enhanced ability to apply course princmles to new areas that was found with 
students trained with the EDAS. The computer m skilled lii ids with careful thought behind its 
design Hid us^ can provide a powerful and effective tool to improve learning in the science 
laboratory. 
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CHAPTER H: THE METHODOLOGY OF THE PROJECT 



Evaluatine The Impact of 
Computer-Based Instruction 
in the Svhoob 

by G. Christian Jemstedt 
and 

TynneD.Camun 



The claims about the inqnctofcomputer-based instruction (CBI)<m students range 
firm die elimination of schools as we Imow diem tod^QT (Piipeit, 1980^ 
cultoe of psychopadis (SardeUo, 1984). The debate about die inqiact of CBI l>as focused on 
both die style in which con^uteis should be used in classrooms and die manner in which the 
outcome firom CBI should be measured (Gumming, 1984). 

That conqwtcr-based instruction fim lead to significant improvement in learning has 
been clearly determined (Kulik, Bangert, & Willirius, 1983; Kulik, Kulik, & Cohen, 1980). 
However, die enonnous varied of CBI products available for instractors and die rich diversity 
m the actual use of con^mters in the classroom have made direct, useful j^jjdications of 
research results difficult to acccmqdish. Evaluation cf die actual, in-use effectiveness <rf CBI 
materials and mcdiods must, therefore, be a regular part of cnqiloying diese tools in the 
classroom. Such evaluations can be called "applied" in diatdicy occur within die context of die 
classroom where pedagogical pnoiitles often must take precedence over evaluation needs. For 
example, random assignment <^ studoits to classes nu^ not be possible in a school where 
traclMg of students by ability is a pdicy. The need for continuous, integral, valid evaluation 
of CBI to(ds and Styles of use is critical if diey are to be understood and used optimally. 
Unfortunately, evaluation, and especially applied evaluation, is a complex enterprise and few 
instructors routinely conduct such evaluations <tfdieir course materials. 

We examined current evaluation techniques and designed a process by which 
continuous, integral, and valid applied evaluations could become a part of die use of CBI 
twhnolog/. TTie rctual project centered around the design and testing of a new computer-based 
educational data ac luisition system (EDAS) for school science laboratories. The intent of die 
prqect was to con^wre die effectiveness of die EDAS with diat of traditional apparatus in die 
performance of laboratory science exercises at die secondary school level. Hie evaluation took 
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place at a regional high school during the fiill academic year, 1985-1986. We worked in 
conjunction wiA the developers of the EDAS» as weU as the teachers and administrators of the 
school, durmg bofli the planning and imirfementation stages of the evaluation. We will use the 
evsluatom actiwties of this actual project as an exanq>le around which we develop the central 
Ideas of how CBI evaluations may be conducted in school settings. Using this actual example, 
a case sto<fy, not only provides concrete iUustrations of the general points being made, but2so 
ensuTO tiiat Ac problems and pitMs of q)ptying theoiy to practice wiU be rcve^ 
considered. The methodology of ^licd CBI evaluation is the focus in the current article: the 
data frwn the particular stwfy by which die general evaluation procedures were tested are 
reported elsewhere (Jemstedt, in press; Jcmstedt & Rothe, 1986). 

THE BASIC EVALUATION DECISIONS 
Decision Making Resources 

Therearemanywrittenresooicestiiatcanbeofsubstantialhetoinconductinean 
iWjfdevalji JuddandKenny(1981)providetiiebestandmostappiopriatcd^ 
offtc bMK issues m such evaluations, though technical sophisticatiOT 

For mag^ «ta<^ore^witii some ex 
Reeman, and Wn^t (1979) and by Isaac and Michaels (1971) provide encyclopedic reference 
mformatoon. T*e classic lesource for jq)pliedevduations is by Cb<* and Camp^ A 
tooa^, philo6ophical, yet direcfly 1?^^ 

descr^on of educational evahiation. Throughout the present article we summarize and 
expand on ideas presoded by diese researehers. 

The Goals and the Type of Evaluation 

«r "^^^^^^^c question we had to ask was why the evaluation was being undertake 

we Identified four reasons for conducting such applied CBI evaluations; (a) to obtam 

knowledge of general mterest about the learning process and CBI materials, (b) to provide a 

measure of ^countabdiQr to educators and administrators who purchase or use CBI materials 

W to guide future movements in the particularprogram we WW 

fim^g to suRXM Amber program development As is ^pically die case, we found elements 

of aU four scmIs m Ae project Phncipalfy, however, we were interested in undetstandinx tiie 

program and descnbing its possible effectiveness. We tiius were emptoying both ofAei^o 

J^raofevab^xm: formative and summative. Our evaluation was formative in that it was 

daected towrds fee<»«ck of information about the prog^ 

The cj^uation was also sommative in that part of our focus was on spec^^ 

ttraifte CTI pB5gnnn m a general sense that couM be en^<yrf 

besKlet science laboratories. We also found that our evaluation was bodi a process and an 

concerned witii understanding die processes that v-rre producing the 
cp.asofAeCTIaswdla8witfiuncovcringthecffect8tiiemselves. Where we did not find a 
blendmg efferent Styles was with the perspective of the stwfy. >^Ued evaluations of CBI 

Diraa tne two different theoretical perspectives of en>k»ation and 
hypogesis testing. Apphed evaluations typically are expkwatory in nature, seeking to find all 
possible effects of the CBI mtervention, rather than to test a particular hypothesis. 

It became clou- to us thai jUUiovgh one can describe in a theoretical sense a hierarchy of 
the vanous types and styles of evaluations (Venezky, 1983), in practice elements of most styles 
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and types tend to be present The issue thus becomes one df making explicit the many implicit 
decisions embedded in designing the actual procedures. We found that evaluation is best seen 
as "system learning," in the sense that it is a process by which a set of individuals in an 
organization kam about Ae conq>lex interaction between their own behavior, the functioning of 
the materials and resources they use, and the behavior of their learners. 

The school is a rich source <rf information; as a natural environment it affords 
researchers the oppOTtunity to evaluate ixograms and innovati(ms in the same context in which 
those programs are to be in^lemented and used. Results obtained in such a setting are well 
suited fot q>plication in o&er, similar environments. While the scbodi provides an advantage 
to researchers in terms of the generalizability of findings, itis alsoadifilcult setting in which to 
maintain control over experiinental procedures. As in odier naturalistic settings, many forces 
odierdian die e}q)aimenter can affect die nianqnilation. The present studty provide an 
exanqiie of how die stnictnre of oqierimmtal design can be superLii^posed on the sch^ 
setting, dierefay preserving die field environmoit and maintaining die coatoA necessary for 
rigoroos scientific research. We remain convinced, as we were when we be^an,diat skilled 
educators can conduct hi^qualitir, valid evaluations as educate. They nuty require 
additional assistants or fimds, dqMmding on die siae of die evaluation, but good evaluation and 
good education can occur simultaneously. Fiirdiermore, we believe tiiat die two must occur 
tt^therif die educatimal process is to maintain or increase its quality and effectivoiess. 

MEASUREMENT ISSUES 

Variables to be Studied 

The choice what variables to sanq>le determines in large part what results can be 
obtained. In an iQ>[died evaluation we maintain diat one must cast a wide net in choosL:g 
variables. We chose variables fiom all duee of die domains diat influence educatiixial 
outcomes: Person, Behavior, and Environment (Bandura, 1978). From die Person domain, 
diose characteristics diat die student brings to die classroom, we sampled die abilities, previous 
school expoiences, personality, and attitudes of die students. In die Behavior domain, die 
actual actndties in which die learner engages, we direcdy observed die students duricg class 
meetmgs and obtained reports firan diem about what di^ did when not in class. In the 
Environmoit domain, die characteristics of die learning environment itself, we reccHded Ae 
attributes each course in^Kliich die CBI materials were used. 

Though it might be desirable to assess students using as many different techniques as 
possible, such a i»x)cedure can oven»4iehn ti?e evaluator widi too great a variety of inf<wmati<Ki. 
We obtained data with most of die different assessnmt techniques diat arc available, but 
concentrated our efforts on four demographic records in die school, student self-reports, 
behaWoral observations, and teacher created performance measures. We occasionally used 
mterviews widi studoits and teachos to collect a supplementaiy set of field notes to hete flesh 
out the <mantitative data during die later reporting stage, but decided diat interviews could not 
provide die objective validity diat die CBI materials required. R>r similar reasons, die student 
self-report measures were obtained widi well-researcheJl, published and locally developed 
questionnaires and did not include open-ended questions toe extended student written 
comments. Furdjermore, widi die natural availability of teacher prepared objective 
performance measures we did not need to enq>loy rqports from significant odiers in die 
students lives to obtain verification of tiieir self-reports. We were especially careful to choose 
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j^bles t^revealed demographic, cognitive, and affective aspects of the students. Since 
jcrc JTO often differences between preferences for and achievement with CBI (Bureer. 1985) 
the distinction brtwjcn cognitive and affective was maintained with the outcome nu^siics as 
pCTfo^Sl how the students perfomcd and how they fdt they 

The Apparatus and CBI Resources 

rFnA«?^?*^"£2??** ^ ^ *^ ^ Educational Data Acquisition System 
gDAS). was dcveloped^a team at Crcaic, Inc. composed of engineers, pswhologists and 
educated with fuiHJmgficni the U.S.Dq^^ -ScSiSfedcsS 
according to sound hunun/act^ 1985) and consists of several 

independently developed conqiooents and a package of software and documentation that 
int^atcstheconqxmcnts. The hardware includes an IBM PC with dual drives serial 
mtatace, pnntCT, and mooochiome monitor, a Bunr-Biown Data Acquisition System with an 
analog to digit^ converter, microprocessor, RAM mcraoiy, and various sienal Wessois- and 
a set of standard sensors fiom a varie^r of manufactured are capaK 
^watare, pressiwe, motion, force, conductivity, sound, light, and electricity The 
ped^opad core of the EDAS was the software. A comprehensive software packafie was 
^^if" ^i*^? ^"^^ in much the same way that a univeSjTSiiiSstry 
rcs^h saentistwodd use a con^tcr. In actual use, the student scientist with the EDAS 

S&'o'fSS^*!^^^"^*^^^^'^'*^"^"^^^ 

«.{ 2S ^ the mparatus ^d software highUghts a phUosophical perepective that 

AdilemmaaSM^Senever^^hastomSea 

treatoente b^g dea^ied and fnted and the control treatments (the status quoTshould om 
^^^^^l^^^.^""^^^ Should 
^!S^.iSfSl H!^l?^ is likcfy to be available to the schools or one that 

^^^^^^^^x^ytohcammc? We maintain in an mpKedevaluationtiterck 
S SSIILSi*^ <~ «*ouid test the best possibl?£SE ^ to 

J^'SL?^*^**'"?*^!?^*^ Optimally each studeiuorpair of staS^ould 

owselves to one con^mter station per classroom. On the other hand, when decidin«wheA» 
tite computer 52«an 8?H>dd a powerfid one ^ 

''^^SS'^"?^^ 0»»«casoningwasthaiv.ewante3^knowh^S 
wouwa»8rt sconce learmng not how a rudimentary computer would affect tlie leamine We 

«1S3^^*^ The first study in an 

applied evaluation, we reasoned, must examine the maximum contrast possible betweai the 

Sf^«?!E?^^ ^ we recSmSend that, wh« 

jSS5fS"***^-^^ cjqwimttital treatment, applied evaluators adopt the perepective of 
n5^?S*^ maxmnun contrast within die limits cf what can actually be aSorapStotte 
natural classroom; m short, create die best, but employ it realistically. " ^ 
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I Research Setting, Subjects, and Sampling Techniques 

The setting of Ac study was a mediom-sized, legicmal public high school located in 
t Hano ver, New Hampshire. The high school was chosen because (a) its students were 

t i^»rescntative of many other small, good high scbods; (b) an agreement could be arranged 

with die administration and faculty to have all science courses use both the EDAS and their 
traditional laboratory exercises, (c) full data could be obtained about students, faculty, and 
„ courses, and (d) the school was near the project headquarters. The population of Hanover and 

the surroundmg community is largely associated with Dartmouth College, an army research 
laboratoiy, and the regional medical center. The high school, however, draws ftom a much 
larger area, representing both towns and small rural communities. 

The high school includes students firom grades nine through twelve. Since the project 
:* was coa^nied with C3I in science leanung,onty the science dei«rtmcnt was invol^^ 

study, llie Ugh school science department olicis six courses in the four traditionally taught 
saenccareas: 1) a genend introductory course in physical science; 2) two courses in 
chemistiy, one general and one advanced; 3) two courses in biology, one general and one 
advanced; and 4) one course in physics with tfiree levels of difficulty. A regular course 
sequence of increasing difficulty is followed by most students. All science courses are two 
semesters in duration. There are six faculty in the dq>artnient,2Q)IH0xiniateIy one for each 
course. 

Chily students who enrolled in one <v more of these six courses participated in the 
study. The total sample size was 461 (209 females, 252 males). Although the mean ability 
level of the sampie is above average, the range in standardized test scores is in keeping with 
n^onal norms. Student participation was considered voluntary in principle; ai^ student could 
refuse to take part However, because the study took place in regularly scheduled classes and 
mvolved the regularly planned learning materials, no students declined participation. Haborate 
measures were taken to protect die confidentiality of die data so diat die evaluators could collect 
detailed information about all aspects of die participants' lives. 

The Relationship of the Evaluation to the School 

The evaluation was designed to preserve die natural environment of die classroom and, 
at die same tinie, to superimpose die structure of experimental control over classroom 
pioccedmgs. To achieve dus syndiesis, we met widi die sci^ce teachers extensively aiid 
contoualj^ duoughout the project The curriculum of each course was examined, and 
laboratories which met dnee critoia were chosen for cxpaha^M con^arisons: (a) die EDAS 
could sense die process in question, (b) die variables involved could be graphed against time, 

and (c) die ^stem could be used in an andogous way to die traditional iqpparatus in 
demonstrating the process or principle in question. 

TTie laboratories chosen for experiments were taken fiiom die regular cuniculum of die 
courses; no new laboratcvies were developed. However, diere was a significant amount of 
activity associated widi pilot-testing labs, which did fall under die rubric of cuniculum 
development Much of diis woric involved adjusting die materials of die texdiook laboratory to 
conform to die specifications of die EDAS and its sensors. Often die quantities of substances 
had to be altered in order to fall widiin die sensitivity range of die sensors. In odier instances 
die substances tiiemselvcs were changed so diat an EDAS sensor could be employed. These 
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however, retained the basic procedures and conveyed the same principles 
as the est^lishcd textbook versions. In addition, any changes in materials that were made to 
accommodate the EDAS also applied to the traditional trcateicnt groups. 

Since part of the evaluation strategy was to employ the EDAS as realistically as 
possibl^we examined two different styles of use. For about half of our experiments we 
?rSfU® computer to assist the teacher in presenting a demonstration lo the class, 

in the other half of the experiments we examined the direct use of the computer by students in 
laboratwy sessions. ITus 50/50 proportion approximated the actual proportion that the two 
styles of use were employed by the teachers. f " "-«nnc iwo 

In ai^lied evaluations it is not always possible to obtain uniform usage of the 
expeiimmtalfi^^ across aU possible conditions in the school. Some tethers were more 

general, it w.« noted SSShSj 
nutolty <9peaied lesist^t toch^^ in their teaching practices and reluctant to take initSS 

*^i^^.?^?*^^^S"*^°^<^»^cula. TWsobscrvakmisswmortS^ 
recent stodia which deiiK»stialB considerable resistance to technological inno^^ on Ae 
pm ^teachers (Sdmmcl. Comwr. McCaskiU. & Dunett; 1981). sSne o^S^cKra^ 

. We counteracted this situation by maintaining a constant presence in the school; 

providing the mitiatove to plan, pUot-test, and schedule labs; assuming aU or most of tte work 

up and testing the EDAS for each lab; verbally reinforcing ^AAe 
teachers the need to conduct many experiments in each course; and offering sevwal forms of 
c<«ipensati<m to both the teachers and th^ Theimportkceof S^caSte 
f^!y^^^-?f-^^^^?^°°^- A semester of wo!k in the year prior to the actual 

SS'^iS^l?'*^"" ^« «>t «^tain Presence in the schodL One 
T**^ changed the classrooms enough to distort the 

♦£5?f "S!^ ^ P*^^ of *e field managi cncoura^ aS 

SSlSv Sl^J? JS?2f ^ ^ they would not have otherwise done, but did not 

^^J^^t^'^^'^'^^l?!^^^^ We noted that teachers^SSngness to 
Sf* unproved after first use and continued to improve over time, as they 

found the woric load to be approximately what it normally was for thSn. 

The teachers' initial reluctance to involve themselves significantly in the project 
tftrcatenol to reduce the total number of experiments and the scope of the evaluationTAlthoueh 
^ iesolvcd,.we were also concerned With the potoSS^disVmtiJf 

?S2!Jl^^"^*^^*«"*"S"""towardiheEDAS. WecountoactedthepossiWUtv 

^ "«f»"y ^'^g *e evaluation's ex^SS rSSSdS 
and uKhcafiM die potential sources of systematic enSr to the teacher before each kb^^ 
addition, if the laboratoy was a teacher demonstration, we observed while the teacher ocacticed 
^25.*^l!f2***^"^ usmg both the EDAS and traditional apparatus during pUot-testing^d v^ 
mdicaled those aspects of the teachers' deUvery which necdSdto be changed in <Sw to Sre 
the presentatiOTs cqmvalcnt Finally, we also maintained a presence in the actual cl-sroZas 
oteavcis; we ^s^id^ the data from labs in which extraneous events caused the ckiisroom 
proceedmgs to duier between treatments. 
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One must exercise great care in dropping data. Since the goal is to obtain an accurate 
picture of what is occurring, when problems during an observed class are likely to provide a 
false view of events, those data must be dropped. For example, if during a class session the 
traditionai or CBI apparatus failed, a fire drill or other such unusual interruption occurred, or 
cndcal comparative data were missing, then one must not included data from that session in the 
analyses. On the other hand, it is crucial that data not be discarded because it appears that the 
results arc not in the expected direction. The discarding of data or experiments must occur 
pHOT to the determinatioo of the relative outcomes from the experiment or serious bias will 
destroy the validiQr of the evalution. 

We often found ourselves preswoing a degree of structure, organization, and adherence 
to schedule that was not evident in the school Some teachers were more organized than 
others, and for some courses it was possible to plan all of the experiments for the year at the 
beginning of the project For other courses, the field manager checked in with the teachers at 
monthly mtervals to plan labcxatories for each time period. 

The planning aspect of each laboratory consisted of obtaining and connecting the 
necewaiy sensors, pilot-testing the actual laboratory procedures with both the experSnental and 
traditional qyparatus, designing the experimental procedures by which to compare the two 
instnictional techniques, developing lab manuals and instructions where appropriate, and 
schcduhng the experiment This planning was directed by the teacher with the fuU support of 
the evaluation fieM manager. Altiiough some teachers rigorously followed th 5ir planned 
cumcula, most courses were significantly behind schedule in their syllabi at the end of the first 
semester aid at the end of the year. The teachers often cancelled laboratories and complete 
sections of the course cumcula in response to these delays. 

It was recognized, however, that the presence of the researchers and the project itself 
were secondary in importance to the presence of the teachers and students and the enterprise of 
education. In all of our relations with the high school, we stressed our understanding of and 

^* *^ hierarchy. At the same time, we emphasized the need for multiple tests of 
the EDAS and for rigor and precision in experimental procedures. In practice, there was a 
considerable degree of cooperation and flexibility maintained by both the teachers and 
experimenters. 

Compensadon to the school 

We noted previously that various forms of compensation were offered to the teachers 
and the school in exchange for their participation in the stu<fy. Compensation was an essential 
feature of the evaluation, since we required use of both the personal resources of the teachers 
and students and the physical plant of the high school. Although we initially expected to pay 
the teachers dircctty for extra time they spent that would not have been required in the normal 
course of events, that became unnecessary. The presence of the fieM manager eliminated most 
of this extra time requirment for the teachers. Compensation to individual teachers took the 
form of increased assistance for the teachers from the field manager. The form of 
compensation for the school involved the transfer of ownership of the EDAS to the school 
upon conq)letion of the project 

In addition, we were able to offer the science department compensation in the form of a 
wealth of information about its courses. For each course we provided data on the effectiveness 
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of the learning materials, the instructOT, and the various instructional media, on student 
perceptions of and attitudes toward the course, and on charav -ristics of students who enrolled 
in the course. These data were obtained from surveys admin: -ed to all science students three 
tmies during the year. 

/ X ?® school's experiencing of the evaluation, itself, also produced other indirect gains: 
(a; considerable curriculum development, planning, and structuring was associated with the 
evaluation; (b) data on the relative merits of instructional iqjproaches were collected and 
analyzed; and (c) the practical experience with evaluation research prepared the school and the 
students for subsequent mvestigations both in-house and by outside evaluators. All of these 
features have simdvcd beyond the term of the project The science department has included in 
Its own budget for 1986-1987 funding for the administration and analysis of the surveys we 
constructed for the present evaluation. ' 

THE RESEARCH STRATEGY 

As indicat^ above, the principal strategy of this appUed evaluation was to mesh the 
rcseittch design and procedure with the curriculum of the courses and the normal functioning of 
the classrooms, to the followmg discussion, we describe the methodology of the study and the 
tactics en:q>loyed m executing this general strategy. 

Experimental Design and Group Assignments 

We applied a relatively novel approach to applied evaluation by designing Uie study to 
be analyzed witii meta-analytical techniques (Glass, 1976; Wolf, 1986). We conducted 
nineteen individual experiments in which Uw EDAS was compared to traditional laboratory 
science instruction. Computations of effect sizes, in additional to statistical significances, were 
wnpl^ed to survey the ovciall effectiveness of computer-based science teaching and to assay 
themnuence of vanous stwfy features on both attitudinal and educational outcomes These 
analyses are innovative for applied evaluations in that they are ayplied to data obtained in the 
same setting, usin^ tiie same subjects, and employing standard instrumentation, design, and 
procedures. We highly recommend them. These procedures offer tiieabiUty to make broad yet 
accurate summaries of a huge body of data. The conclusions can be stated more cleariy and 
simply than can the results of 19 separate experiments. Fmally, tiie meta-analytic, quantitative 
summaries are more hkcly to generalize accurately that would a veri)al summary of Uis 19 
dinerent sets oi results. 

In an appMcd valuation the choices among tiie three factors that differentiate between 
research designs gudd & Kenny, 1981) are fairly obvious. The first factor, how learners are 
assigned to the i^erent treabnents to be tested, is typicaUy determined by the existing class 
arrangement TTje second factor, whether a pre-tieatinent measure of tiie subjects is taken 
simply must be t put of an appUed evaluation because of tiie many forces preventing random 
assignment of stiidents to tieatinents. The tiiird factor, whetiier 911 participants experience aU 
treatments or cniy some, is best handled by repeating tiie treatinents enough times tiiat aU 
participants do receive all treatments. The meta-analytic procedures descnbed above are quite 
consistent witii tills decision. ^ 

For tiie present study, two treatment groups were established for each experiment, one 
usmg traditional apparatus and procedures and one using tiie computer-based EDAS approach 
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in the performance of laboratoiy exercises. The sectional divisions of each course were 
followed in fonning group assignrosnts for the 19 experiments. We randomly assigned use of 
the computer to one to duee sections and use traditional apparatus to one to three sections, 
with the total number of treatment groups dependent on the number of sections in each course. 
In some experiments, however, tiie lengtfi of the exercise prohibited use of the computer for 
more than one lab group (4 to 6 students) within each section; in these instances, we randomly 
assigned use of the EDAS to lab groups. This is an example of turning a prescribed condition 
m an aj^Iied evaluaf ,a into a design feature. By so vaxying the manner in which lab groups 
access^ the computer, we were able to make an additional, unplanned comparision, between 
two dirferent, realistic ways of employing die computer in class. 

Because most students follow a regular course sequence in che science curriculum, 
class year was a major determinant of enrollment for each course. In addition, ability level 
conuibuted to course selection and to sectional placemoits within a course. Although the 
science courses lacked a sufficient number of aWlity-iielected section-; to create a highernmLr 
desi^ (treatment by abiUty), we included steps in the analyses to check for the potential effects 
of ability on the treatment outcomes and control for ability as a possible confound throueh 
analysis of covariance. 

Lab groiqungs weic teacher-selected at the beginning of the year. Although method of 
groupmg has been shown to affect performan .« in and attitudes toward Jaboratofy science 
courses, we did not attempt to mampulate this variable in the present evaluation. The teachers 
altwed lab group assignments on some occasions throughout the year, but these decisions were 
made without respect to the evaluation. We made no changes in the established erouoines for 
experimental purposes. f e> 

Student Data Measures 

Our goal in collecting data was to use standard, validated questionnaires and tests 
wherever possible. Table JL 1 presents an overview of the data that were obtained about the 
studaits. The standardized test scores, giades and demographic profiles of the subject 
population were obtained fix>m the transcripts on file at the high school. The standardized tests 
mcluded the MetropoUtan Achievement Test, PSAT, SAT Aptitude and Achievement Tests, 
and a national test administered only to sophomores. Not all students took all tests. 
Throughout die data anafyses we never estimated missing scores, instead for anv given 
con5)arision we employed only dioses students for whom we had full data. 

Grades from all previous course woric in English, science, and madiematics were 
obtained for each student No achievement data were available for die current year, except for 
die students' semester and final grades in dieir science courses. This situation meant diat die 
only achievement data available for fioeshman subjects were current science course grades. 

DemograiAic information collected for die evaluation included class year, year of birth, 
birth order, fanuly size, gender, junior high school, and number of years in attendance at die 
hgh school. 
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Table ILl 
Variables and Data Measures 



Domain 



Person 



Behavior 



Environment 



Outcome 



SouiccoftheData 
school transcript 
self-report inventory 



daily class questionniare 

observatimal checklist 
interviews 

school records 
self-report inventory 

interviews 
teacher graded woric 



Measure Obtained 



standardized test scores 
courses taken 
previous grades 

demographic information 
personalis 
study habits 
learning style 
attitudes towards school 

attitudes towards class 
attitudes towards con:q)uters 



on- and ofT-task activities 
actions during class 

course characteristics 

instructor characteristics 
course characteristics 

instructor chai^'^^^tics 
course characteristics 



lab reports 
quiz scores 
examination scores 
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Based on previous psychometric wofk and extensive pilot studies, we constructed a 
self-iqKnt inventory using 5-point scales to assess students' learning style, study habits, 
academic and social self^steem, and satisfection with their school pafoimance. A number of 
the personality questions were directed towards self-efficacy, since perceived self-efficacy may 
be an important aspect of school achievement (Schunk, 1984). In addition, the inventory 
asked students to evaluate various aspects of their course, including instructor characteristics, 
grading criteria, and overall (quality. The original version of this survey was composed of 160 
questioos; this form was adnunistered during die pitot-testing i^iase prior to die actual 
evaluation and factored to the current form, which consists of 69 questions. TTiree versions of 
die mvo^jry were established, all of which were equivalent in form and questions asked, but 
which differed in terms of lengdi and timing. One version asked students to answer accoiding 
to what diey expected their science course to be like for the current year, one asked students to 
answer accoiding to what diey had experienced in their sdence course during die te^ 
and one asked students to answer according to what diey had experienced in die science course 
durmg die second semester. The last two versions included 41 additional questions which 

adced students to nue die various instrucdooal media and course rnaterials on three dimensions' 
usclutoess, difficulty, and integration widi die rest of d» course. TTie pilot phase of die project 
before die actual year of evaluation began was of major value. We were able to local many 
S'Hirces of difficulty and improve many procedures before tii^ could jeopardize the actual 
evaluation. 



Treatment Outcome Measures 



A daily report questionnaire was ccmstructed using 5-point scales to assess pre- and 
post-clasi attitudes in the experimental lab sessions. On these daily reports students were 
asked to ntt tbek enjoyment of die class, tiieir interest in die subject matter, die difficulty of die 
cl^?/4hcir enjoyment of using computers, aid dieir level of comfort in using computers. In 
additioii, two unique questions were asked on die pre-class form to assess before-trcatment 
motivation to learn and to do well Six such unique questions were asked on die post-class 
survey: die amount of knowledge learned in diat da/s class; students' confidence in dieir 
abuities; the effectiveness of the mediods used to leam; the students' certainty widi die 
procedures; die originality of their woric; and die hurriedness of diat class period. 

An observational checklist for die assessment of teacher and student behaviors was 
developed widi extensive pikK woric and used in die class labs. Three categories of teacher 
behaviors were included in die checklist: off-task to all students (e.g., out of die room); 
providing individual attention; and providing gcovp attention to all students (e.g., lecturing, as 
m a denionstr>Hjn). Observations of teacher behaviors were recorded at one-minute intervals. 
The three categories of teecher behaviors were considered mutually exclusive; only die single 
category most displayed during an observational period was recorded. Six types of student 
behavior w«B included in die checklist: off-task; group-oriented (in die performance of the 
task); mdivkhially engaged with die instructor; attending in a group to the instnictor, attending 
to die iqjpiratus; and note-taking. Estinutes of the proportion of the class engaged in each 
catcgoiyofbehaviorwcrerecordedaftercachsuccessiveminuteofclasstime. Typesof 
student behavior were not considered unique; die same student v/ho displayed f vo types of 
bdiavior diuing a single interval was counted in die total proportion for each cawgory . The 
proportions cf die class engaged in all cate^cxies in an oMervational period couW sum to more 
dian one. Five fiactions of class participation were used by die observers: 0, representing no 
students; 1/4, representing from 1 student to 1/4 of die sample; 1/2, representing more dian 1/4 
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Of the 8ai!q)le but not more than 1/2; 3/4, reFresenting more than 1/2 of the sample but not mons 
than 3/4; and 1 , representing more than 3/4 of the sample. 

Assessment <^ acade m ic peifonnance wss made by obtaining teacher-graded reports, 
quizzes, and exams. No special performance measures were constructed; only the the 
assignments included in the normal grading process were analyzed Data prepared for analysis 
mcludec scores on each qvestioo of a particular exercise that related to the students' work 
during the laboratory; subscak scores formed by summing the raw scores of all related 
questions; and the total scores of each exercise. The subscale grouping:^ were cieaied to 
classify student performance dc juuns: Act recall, use of principles, and appUcation or 
extrapolation of information to new situations. Variables such as these subscale groupings 
have been labeled measures of scientific literacy (Arons, 1984). ^ r o 

Procedures 

Tte field managa served as experimenter throughout the evaluation. Not only did this 
pvc hmi full conttol of all procedures and ensure the maintenance of good scientific methods, 

It also encouraged the students to assume a more careful manner when filling out forms, as his 
presence came to represent for them a thoughtful self-reporting of their states. 

J^sequcnce of procedures foUowcd during the evaluation is iUustrated in Figure tt 1. 
The self-report mventoriss were administered in each course during the first two weeks of the 
first seinester, in tte to two weeks of the second semester, ard in the final two weeks 
academic year. Admrnusralions were made in individual dus meetings. For all 
administrations, the experimenter read standard instructions, part of which cited examples of 
how student responses were taken seriously and were nuddng a difference in the way me 
courses were tau^t and organized. These statements to the students have a positive effect and 
can significantly improve the accuracy and validity of the self-icport measmcs. 
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Participant 



Project Phase 



Dssign Pilot Study 



Main Study 



Sapt Apr Aug Sept Jan May 



Activities 



student 



t 



t 

u 



t 



fill out full set of 
preliminary questionnaires 
f complete personal questionnaires 

J particijMite in classes (both 
traditional and expenmental) 



teacher 



establish liaison with evaluator 

plan experiments with 
comparisons between 
treatments 

J teach traditional and 
experimental classes 



evaluator f 



t 



prepare questionnaires 
decide on treatments 
plan treatment comparisons 
select population 
establish liaison with school 

conduct pilot studies 

redesign materials and 
planned comparisons 

J conduct comparisons 
between traditional and 
experimental treatments 



I Key: f 



single occasion 



J continuously | 



Figure HI. The ten^xyral sequence of activites engaged in by each of the particioants in the 

applied evaluation. 



CREARE PROnUETARY TS-ill 

ireare 

Bcponncnlal comparisons of the two 
conducted tbrougMt the academic year. These experimental labs foUov»«d the regular topical 
sequcnos and schedules as established in the course syllabL A standaid procedure was 
^^Jia^^i^^ expsrimcnt After attendance was taken by the teacher, the experimenter 
addiwscd the class, mdicating that as part of the on-going evaluation of their science course, 
tte students were bemg asked to fiU out two questionnaires in fl^ 
beginningandonc^end. To prevent Ae experimenter's presence fiombei»g strongly 
^sociated with the EDAS, so that students eqjected to use the system in particuL^ 
ffii expnimenter^evalatfed class meetings in which experimental co^ 
bOM^ttn. Inoriertoprevcmexpectan(ycfifectsfom<&tortingthepre<lassdailyrci)o^ 

an stt^ Once aU students were finished filling out the 

of^K*«andsti^tbehivK». In those expcnments for which sectional grow) assignments 

JS2?5-2S±^J^.??i^^ Ihthoseex^Sntsf^S 
trwtacnt Avisioos w« made by lab groups th^ 
ntoyalotacrvaaons be^«tt 

jjjmflie first lab group finished wiA the labon^ 

am t&e post-cIaM questionnaires were handed out to each group of students as diey finished. 

ot ffieSJAS posedaprobten. On some occasions the i^aratus developed technical nroblems 
Jjitethestodentswwpcrfoimingthelaboiatoeyexer^ Inmost<^theSSor^ 
teachercouldconectAeproblem. For major problems, however, the eimerimenter was the 
only person present with sufficient knowlttjge of the system to attend to them. If these 
FoblOTS were easily and quickfy corrected, the data fiom that section were retained; however. 
If the problem necessitated prolonged and involved attention by the experimenter, the data fiom 
that particular section were discarded. Though these problems were inficquent. having a 
protocol pi^ared to deal with them was important for minimizing interference with both the 
teachers and the evaluates s goals. 

Data Processing 

*^^JP^ prciMration ^data for a Ms involved converting each raw set of Jata into a 
standaid matox foimaL We emptoyec variety of multivariate data analysis procedures to 
SSSStl!l5*JT*T! »fbi«^.that provided a high levd of interaction between 

mvwtigator and data. ItisveqreaiytDlose sight of thedataina complex evaluation due the 
sh^ number of numbers. TTic use of interactive statistical packages maintains a closer contact 
wia the numbers being m a nip u lat ed. Our policy was to take the conservative approach. 
Where multi^ tests woe made, wc adjusted significance levels to reflect the mmy tests being 
performed We toted aU assumptions about distributions of data that were requi^ 
vanous stadstic^ tests. We graphed data fiequently to direcdy observe the shi^e of effects we 
wwestu^g. We strongly urge a similar approach for other investigator. A large data base 
and powerful imalysis techniques are seductive to the evaluator, conservative proc^uics can 
prevent ansdysis from becoming a fishing exptt''*«on for results. The goal must always be that 
of discovenng the patterns that exist in the dat;:, not in proving certain hoped for findings. 

We expected that wcwouW need to covaiy academic ability in many of our analyses 
because our outcome measures involved traditional lab reports and examinations and because 
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some classes were stratified by ability. Our inteat in collecting die many diffeient standardized 
tests was to find an objective measure of ability diat could be used as a covariate in the analysis 
ofdietreatmratotttCOTies. To this end we created a single aki% matrix by ombining all 
SQMitude and achievement sttbscales in a single matrix. Hiis matrix was dienfiu:tor analyzed. 
The resulting rotated factxx pattern revealed a diree-tiered hierarchy, ace hunting ftH* 70% of the 
total variance. Hie fectors were (a) general mental ability, which accounted for 50% of the 
variance; (b) demonstrated achievonent (15% <^ variance), best represented by cumulative 
class radc; and (c) specific mental ability (5% of variance), in die area (tf madiematical 
abstraction. We chose die semester grades in die science course as the best represoitative of 
die general inental ability &ctor and used it as covariate in die analyses. 

Widi die observational cheddists for each cat^ory of student behavior, we counted die 
total number of times each of die five possible fractions of class participation (0, 1/4, 1/2, 3/4, 
l)occuned. Wealsocoanteddiefiequencietofeachcategoiyofteacha>behavior. This 
procedure was p er fo rmed on the data for each treatment groq> in each section of die course. 
The data <tf CQiiiinoa treatinent groqK were then surmiwd across sections, producing a 2 X 5 
frequency matrix for each category of student behavior (treatment by proportion), and a 2 X 3 
frequency matrix <tf teacher behavior (treatmoit by cat^ory). 

CONCLUSIONS 

The results of die evaluation were most illuminating. As mentioned above, diese 
results are summaried in Ou^pter nL 

For die presoit pqier our goal was die undostanding <rf^ die q>ldied evaluation process, 
itself. We conclude diat sophisticated ^liedevaluati(»is are possible in die scho(^ The goals 
of die evduator and die gods of die educator can be met simultaneousty, widi little serious 
compromise required for eidier. fri fact the educatw and evalaator can enhance each nthftre 
objectives. This omclusion is b(xne out by die bdiaviorc^ die teachers subsequent to die 
evaluation. The school is actively pursuing evaluatim of odier aspects of die science 
curriculum and odm dqxotments are inquiring about how th^, too, can collect such data. 
This frunlitati(» of on-going evaluticm i»actices win also make it easier for odier researchers to 
work widun die school Thus not cmly can die goals of die applied evaluation be met, but a 
style of curiosity and effective evaluation can be established in die school for die future. 
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CHAPTER in: RESULTS 



A Meta-Analytic Experimental Comparison 
of Computer-Based and Traditional 
Laboratory Instruction 

by G. Christian Jemstedt 



Though the claims and cxpectadons for computer-based instruction (CBI) benefits in 
educatiott have cften been excessively great, it is cl^ that computer-based teaching can have 
positive etiects on students' attitudes and learning (Kulik, Bangert, & Williams 1983) 
Research that is directed at uaderstanding the nature of the impact rather than whether it exists 
or not now sems to have the most appropriate emphasis, fhe need to understand more fully 
the ur^tof CBI w espcciaUy acute in science learning for a mmiber of reasons. First, there is 
HtQe CBI research that has been directed specifically at the science learning area. Furthermore 
computers a?ie particulariy important tools in university and industry science research so there is 
a grey deal of pressure on science teachers to integrate computers into their curriculum The 
use of conq)'itm m science teaching also appears to be an especially promising way of 
canymg outMachr's (1983) iccomnwndatico that science instruction help students develop 
mdcpcndence m their learning, become confident of their abilities, and function well in 
enyircnmoits when tficy can colore on their own in ways similar to the ways that 
^fcssional scientists use. FinaUy, it is clear fixjm the analyses of the National Assessment of 
Educational ^ogress data (Walberg, Pascarella, Haertel, Junker, & Boulanger, 1982) that 
classroom morale and direct instruction fix>m the teacher arc the key variables in producing 
positive m the classroom. Since CBI has been impHcated in both improving mor^e and 
enhancing direct instruction, it is a potential tool for inroroving science education that must be 
more fiUIy understood. The pw^cnt study is directed it refining our understanding of 
computer-based instructicm in the science classroom 

'I^m5>act of CBI on educational outcome can best be conceptualized in terms of a 
mrec clemait model of human behavior that ascribes instructional outcome to the joint effect of 
three domams of mfluence, the jichaaa of the student in the learning situ on, the cognitive 
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and affective internal states that the student as a pgisca brings to the classroom, and the 
— " at of the classroom itself (Bandura, 1978, Jemstedt & Chow, 1980). 



Li woik relevant to the person domain, Kulik, Kulik, and Cohen (1980) reported that 
attitudes and perfonnance were not found to be directly linked in studies of CBI. In some 
studies attitudes imptovcd while perfonnance did not In other studies performance improved 
but dropout rales also increased. Kulik, et al. (1983) found improvement in both performance 
and attitude, but the attitude results were relatively smalL The intact of CBI on attitudes 
towards science is a matter of considerable inqwrtance, as attitudes determine fiiturc motivation 
towards science (Tamir, Welch, & Rakow; 1985) and there is evidence that science attitudes 
doCTcase as studente in secondary schools have more exi>08ure to science (Hofstein & Welch, 
1984). bmce attitudes m science courses are multidimensional, and include attitudes towards 
the course and its scientific discipline, the teaching methods employed in the course, and 
actual class activities (Okcbukola, 1985), the present study coUccted a variety of attifadin 
data. 



the 

if:adinal 



p» the behavior domain, time on task is the principal variable that seems to predict 
acad<9iucachievenwnt(Seifer& Beck, 1984). Level of task engagement has a strong impact 
on achievement and is, itself, influenced by teacher behavior (Tobin, 1984). ftevious work, 
however, has failed to isolate the particular conqwnents of laboratory behavior that are most 
directly related to achievement gains. Okebukola (1985) reported that practical skills such as 
m a n i pu lation of the apparatus were important in understanding higher level, more cognitive 
outcomes. The present stu^ included a set of behavioral measures of bodi teacher and 
student These measures were specially developed to isolate the partic?ilar aspect of the 
laboratory activities that were most directly related to achievement However, the direction of 
causal impact was not assumed to be from behavior to cognitive outcome. Rather, the 
behavioral and the cognitive outcomes were each considered to be separate, though probably 
related, conqxments of the science leammg process (Hacker, 1984). 

The achievement outcomes from the science learning were also approached in the 
present study as multidimensional processes. In light of previous work, the performance tests 
woe dmdcd mto toee categories of performance: fact recall, use of principles, and ajpUcation 
abihty(Nicolson,Bowen,&Nicolson, 1984; Tamir, 1985). «vf " 

Fot both practical and pedagogical reasons, computers have been employed in two 
different modes m science laboratoiy instruction. The most obvious mode is that in which 
students diemselves use the computer, in place of tiaditional tqjparatiis, to ccoiduct 
cxperimente. fin contiast to this mode of active stiident use of the computer is the insteuctor led 
demonsttation. In this mode the instiiictor employs the computer, in place of traditional 
apparaftis, to conduct a demonsttation which the stadents obseive but do not physically 
participate hu The lilerattire on mode of apparatiis use, whether the apparattis is a computer or 

investigators find no difference between the two modes of use (Bates, 
1984). However, others have iraorted relevant data that smaUer classes are associated with 
higher achievement (Smith & Glass, 1980) and that the laboratory must play a cental role in 
science teachmg (Welch, Klopfer, Aikenhead, & Robinson, 1981). 

Of particular importance to the nwde of use of the computer is the teacher's behavior. 
Class size, per se, may be less important than the teacher's actual activities with stiidents 
during class (Rosenshine, 1979). Ftactically, the demonsttation mode is easiest and the least 
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«q)ensive mode, smcc only a single computer and one trained user is required It is likely that 

economy may encourage teachers to adopt the demonstration mode as the 
preferred Therefore it is m^xnlant to resolve the issue of whether one or the other mode is 
more effective in its in^t on the educational outcome. Iti the current study, both modes were 
en^lryed and compared Furthermore, for the treatments where students actively used the 
computer, class size was systematically varied in order to separate the mode of use effects from 
the class size effects. 

^ ^2 u"^-",^ apparent in their results, there is a suggestion in the meta-analysis 
conducted by Kulik, Kulik, & Cohen (1980) that the novelty of computer-based instruction, in 
addition toMtoe actual mstructional effect of the computer, m^ have influenced the quaUty of 
icammg. Tlie issue of previous experience with coL^tcrs has not been directly examined for 
Its possible inq»ct on the use of CBL Yet differences in degree of experience with computers 
are large across t^hers as weU as across students. 'iTie present study examined degree of 
eiqwience of teacher and students for its impact on the effectiveness of both traditional and 
conq>utBr-based instruction. 

A vari^Jc that cmeiged in the fieW notes during the pUot studies that pi jcedcdt^ 
experiments of the present study was time of year. The behavior of the students, in terms of 
attcntooo to laboratoiy woric and motivation for stutfying, dropped dramatically as the academic 
year (toew to a close. Teachers anticipated this by avoiding standard laboratoiy woric near the 
25 semester. Since CBI influences time on task, the possibility of an interaction 

between CM effectiveness and time of year must be considered Moreover, since many of the 
teachers la die current stu<fy were not highly familiar at the beginning of thr year with the 
computer mcAods that they were employing and had their students employing, the possibility 
of a novel^cffect existed AccDidingly, Ae effects of the CBI treatment were separated in the 
present stu^ into twotime periods: the beginning of the year when novelty and unfamiliarity 
might be affectm J performance, and the middle of the year, when the end of year decay had not 
yet reared Its head and the novelty of startup had diminished 

A very unfortunate and confusing feature of much of the woric on the impact of CBI 
has been the confcundmg of the media, conq>uters, with the mode of use. Meta-analyses of 
diffoent media have shown positive effects on educational achievement for most of the major 
moto techmques that have achieved popularity (Rntrich, Cioss, Kozma, & McKeachie, 
1 V86). How^si, as Pmtnch, et al. point ou*, a problem in this research has been that the 
method and the media of instruction have been confounded, perfiaps because of reporting 
P"*f*T^ °^ because it is difficult in some instances to distinguish between them. The 
methodof usmg the computer was unconfounded fix)m the media in the present study in two 
ways. The software developed with the computer system was specifically designed to be 
gcncnc, m the sense that it did not require students or teachers to follow a fixed foraiat or 
cumculiun. Rather the software presented a menu of tools that could be employed in any way 
the user desired, so that the user, not the computer, determined the methods. Secondly, 
teachers and students used both traditional apparatus and computer-based apparatus in 
cooductmg theur woik. Each participant experienced numerous occasions of using each media. 
This combinmg of all subjects with aii treatments was a major feature of the design and 
something that has been missing in too much ef the previous work on CBL The goal of 
separating the computer, as media, ftom tlie methods by which it was used appeared to be 
accomplished; m the laboratory, students were observed to be using, at one time or another all 
fourof the classifications of type of computer use that Rushby (1979) has made: instructional 
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revelatory, conjectural, and emancipatory. It should be noted that the one methodology feature 
that was built into the con^uter system was to make it visually based, both because of the 
value of visual-based instruction (Cohen, Ebeling, & Kulik, 198 1) and because the automatic 
graphing of data by the computer seems to be a major use and value of computers in the science 
classrconL 

Previous work on CBI has focused far more on tlie computer's impact on the students' 
behaviors than it has attended to the teachers' behaviors. However, many classroom variables 
related to teacher behavior influence science achievement as much as student ability (Tamir, 
Welch, & Rakow, 1985). Some work indicates that CBI is effective because it allows the 
teacher to provide more individual attention to students (Evans, Mickelson, & Smith, 1984). 
One must be careful, though, not to ascribe a unidirectional impact of teacher behavior on 
student behavior, the in^iact of student and teacher behavior appears to be bidirectional 0>oyle, 
1979). In the present study both teacher and student behaviors were examined for their mutual 
impact on the educational outcome. 

The goal throughout the decign of the present study was to cast a wide net in an effort 
to detect the full richness of the processes influencing the impact of the computer on science 
learning. Coupled with the wide net was a conservative statistical approach to guard against 
overgeneralization and the quick but unreliable or invalid answer. 

METHOD 

Subjects 

Four hundred sixty one students (209 females and 252 males) smolled in six laboratory 
science courses at a regional high school participated in the study. Students in grades nine 
through twelve were approximately equally represented in the sample. 

Treaiments and Conditions 

Because most students follow a regular course sequence in the science curriculum, 
class year was a major determinant of enrollment for each course. In addition, ability level 
contributed to course selection a>^ to sectional placements wthin a course. The hi^ school 
scierice department offers six courses in four basic areas: I) a general introductory course in 
physical science; 2) two courses in chemistry, one general and one advanced; 3) two courses in 
biology, one general and cne advanced; and 4) one course in physics with three levels of 
difficulty. 

The stu^ employed two different types of laboratory apparatus; the largely mechanical 
apparatus traditi(nially used in the classes and a conpiter-based educational data acquisition 
system. Bofli the computer-based and the traditional apparatus were employed in two different 
rriodes of class meetings. In 11 of the laboratories the teacher was the user of the system, 
givmg a demonstratiim to the assembled class. In the other 8 laboratory sessions the students 
cmplpyed the computer-based system directly. For both different modes of use, the sectional 
divMion ; of each course were followed in forming group assignments for the experiments. We 
randomly assigned use of the computer to one to three sections and use of the traditional 
appaatus to one to three sections, with the total number of treatment groups dependent on the 
number of sections in each course. In 5 experiments, however, the length of the exercise 
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pn^bited use of the conspvter for mwe thai one lab group (4-6 students) within each section; 
in these instances, we randomly assigned use of the computer system to small lab groups rather 
than to the whole section. 

The degree to which computers were used in other ways during in each course was 
iccorded and converted to a varisile indicating the combined instructor and student degree of 
experience with computers. Time of year was also cooed, either as the beginning of the year, 
when teacher and students were becoming familiar with procedures and each other, or as the 
middle of the year when procedures had become familiar and regular but the end of year 
malaise and loss of attention had not yet occurred. The start of year period corresponded 
roughly with the first half of the first semester, while the middle of the year corresponded 
roughly to the second half of the first semester and first third of the second semester. 

Apparatus 

The computer-based educational data acquisition system (EDAS) tfiat was used in this 
study is a laboratoiy computer system developed with funding firom the U.S. Department of 
Education by Creare, Inc. The EDAS omsists of several components, including an IBM PC 
dual drive computer, a Burr-Brown Data Acquisition System, and a special' constructed 
software package. A variety of physical sensors (pH probe, conducti vity meter, pressure 
transducer, thermocouple, accelcrometer, optical density sensor, and light source) connect to 
the system. The dependert variable measured by the sensor is grariied against time, with the 
rate and total time of sampling positioned along the x-axis of the video display. The system is 
modular, new sensors can be mtegrated by inputting the conversitm ratio and other 
specifications of the sensor. User input is by single-character key !x)ard commands during data 
collecuon. The entire system is menu-driven, usmg directional keys that control a movable 
highlight 

Measurement Instruments 

The organization of the measurement instruments used to obtain data about the 
participants in the experiments is indicated in Table m. 1 . Based on previous psychometric 
woA and extensive pilot studies, a self-report inventory was obstructed using 5-point scales 
to assess students' leamine style, study habits, school activities and attitudes, personality 
(includrng motivation, self-esteem, and social behavior). The inventory also collected other 
demographic infomiation. In addition, the inventory aslnd students to evaluate their course on 
a number of dimensions, including instructor characteristics, particu^'x features of the course 
(e.g., lectures, labs, audio-visual instruction, computers used in lab , and overall quality. 
The original version of this survey was compost of 160 questions; this form was 
administered during the pilot-testing phase (the spring semester preceding the 85-86 ac ademic 
year) and factored to the current form, which consists of 1 10 questions. Three versions of the 
invcntoiy wjac established, all of which were etjuivalent in form, length, and questions asked, 
but which differed in terms of timing. One version asked students to answer accwding to what 
they expected their science course to be like for the current year; one asked students to answer 
according to what they had experienced in the science course during the first semester, and one 
asked students to answer according to what they had experienced in the science course during 
the second semester. 
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Measureme nt Instruments 



Domain 



Source 

of information 



Interval of 
measurement 



Variable being 
measured 



Person 



Student 



Semester 



Learning style 



Study habits 
Schdd activities 
School attitudes 
Personality 
Demographics 



Enjoyment 

Difficulty 

Interest 

Attitudes towards computers 
Enjoyment 
Comfort with 

MotivatLcm 
Toleam 

To do well in course 

Attitudes towards lab 
Knowledge gained 
Confidence in abilities 
Effectiveness of mctiiods 
Know what are doing 
Independence 
Rushed 



Person 



Student 



Laboratory 



Attitudes towards class 



Person 



School 



Year 



Courses taken 
Grades in courses 
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Table mi (continued) 



Eomalti 



Source 

of information 



Interval of 
measurement 



Variable being 
measured 



Bchavi(»: 



Observer 



Laboratory 



Off-task 

Group involvement 



Individual attention from teacher 
Attention to teachtt by group 
Attention to the apparatus 
Writing /note taking 



Fact recall 
Graidiing ability 
Use of principles 
Application ability 
Total score 

Examination 
Fact recall 
Use of principles 
Application ability 
Total lab question score 
Total exam score 



Behavior 



Teacher 



Laboratory 



Lab report 




52 



€reare 



CREARE PROPRIETARY 



TN-411 



Table in.l (continued) 



Domain 

Environment 
Environment 
Environment 

Environment 

Environment 



Source 

of informatim 



School 

Teacher 

Student 

Student 

Experimenter 



Interval of Variable being 

measurement measured 



Year Demographics 

Year Course characteristics 

Year Student expectations 

for the course 

Semester Course characteristics 

Instructor characteristics 

Laboratory Science discipline 

Degree of con^uter experience 
Time of year 
Mode of use 

student active use 
instructor led demonstration 
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A daily iq)<)rt questionnaire was also constructed to as%ss pie- and post-^^ 
in the laboratory sessions. As with the inventory, thes^ daily ^cporf^ employed 5-point scales, 
where one extreme represented full agreement and the other extreme rcpi\jsented no agreement 
On th^ daily reports, students were asked to rate their enjoyment of the class, their interest in 
Uie subject matter, the difficulty of the class, their enjoyment of using conmutei?! «id their 
oegrfcc of comfort with using computers. In addition, two unique questions were asked on the 
preHjlass form to assess bcfoie-tteatment motivation to learn the material and to do well in 
class. Six such unique questions were asked on the post-class survey: tfie amount of 
Imowledge students felt Uiqr had learned in tiie class; tfie students' degree of confidence in their 
abihucs dunng class; tfic effectiveness of die mediods used to leam ia class; the decree to 
wbch stadcnts' felt they toew what tiicy were doing in class; tiie degree to which students 
foUpwed directions or woriad indmendentiy; and tiw extent to which students felt rushed 
dimng the class penod. Teachers fflkd out a similar focm after ti»e class meeting reporting on 
what they thought class was like for the students. 

An observational checklist for tiie assessment of teacher and student behaviors was 
dcvelc|)ed with extensive pik>t work and used in tfie individual laborat^ Three 
categories of teacher behaviors were included in tfie checklist off-task to a!l students (e e . out 
of room); providmg individual attention to students; and providing attention to groups of 
studentt (ej., Icctoring). Observations of teacher behaviors were reco«ded at we-ininute 
intervals. Hie tfute categories of teacher behaviors were considered mutuaUy exclusive; only 
tiie smde cat^ory most display^ during an observational period was recorded. Six types of 
student behavior were included in die checklist: off-task; on-task group involvement iii 
laboratory activities; receiving individual attention finom tfie instructor, attending as a group to 
tfiemstnictoR atterKling to tfie apparatus; and note-taking or otfier forms of writing. Stirnates 
of the proportion of students engaged in each category of behavior wc*^ recorded after each 
successive mmute of class time. T^pes of student behavior u-ere not considered unique- tiie 
same student who engaged in two types of behavior during a single interval was counted in the 
total proportoon t)f each category. In otfier words, tfie proportions of students engaged in all 
categories of behaviors did not sum to one. There was considerable overlap betwera 
categories. Fcr instance, stiidents who were receiving individual attention ftom tfie instructor 
were sometimes writing at tfie same time. However, when tfie instructor was on-task to tfie 
entire class, as during a lectiire, no students were construed to be individuaUy engaged witfi 
tiieir teacher. Five proportions of class participation were used by tfie observers: f 
repiescntmg no students; 0.25, representing from 1 student to 1/4 of tfie sample; 0.5, 
rrawCTting more tfian 1/4 of tfie sample but not more tfian 1/2; 0.75, representing more tfian 
1/2 of tfie sarnple but not more tfian 3/4; and 1, representing more tf'an 3/4 of tfie safimie. One 
highly tramed expenmenter recorded all behavioral observations. 

Assessmertt ci acade mic performance was made by obtaimng teacher-graded reports, 
quizzes, and examinations. No t^pecial performance measures were constructed; only tfie 
assigmnents included in tfie normal grading process were analyze Data prepared for analysis 
incuded scores on each question of a particular exercise tfiat related to tfie students' work 
durmg the experimental lab sessions, categorized into fact recall, use of principles, or 
apphcations of material to new situations. Subscales formed by summing tfie scores on all 
laboratory related questions and by summing all scores 0.1 tfie total exam were also created for 
each exercise. 
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Pi:ocedure 



Experiments were conducted over the M academic year 1985-86. The self-icport 
inventoncs were administered in each course during the first two weeks of the first semester in 
Uie first two weeks of the second semester, and in the final two weeks of the academic year. * 
These mventories were administered in individual class meetings rather Oian in large group 
meetings of all sections oi the course. For all administrations die experimenter read standard 
mstructions, part of which cited concrete examples of how student responses were being taken 
senously and were making a difference in the way the courses were taught and organized. 

Experimmtal conq>arisons die two laboratcny instructional techniques were 
conducted diroughout the academic year. The regular topical sequences of each course and 
ej^jounaits were followed and experimental comparisons were conducted onty with reeuiarly 
sc^duted laboratory exercises. To avoid detrimental novelty effects due to teacher 
unfamihanty with Ae computer system, we worked in conjunction with the teachers in 
plannmg and pilot-testing those laborataies in which use of the computer was pcible The 
conaputer wm only enq)loyed for laboratories in which three criteria were met S the system 
could sense Ae process m question, (b) the variables involved could be graphed against time 
and (c) tfie system could be used in a way analogous to the way the traditional apMratus was' 
used m demonstratmg the process or principle in question. 

The computer-based system was compared to the iradirional apparatus in 1 9 
laboratones dunng the course of the year. A standard procedure was followed in each 
cxpaaatnt. After attendaice was taken by the teacher, the experimenter addressed the class, 
indtcating tfiat ^ part of the on-gomg evaluation of their science course, the students were 
bemg aslred to fill out two questionnaires in that day's class, one at the beginning and one at 
the end. Smcc the experimenter's presence may have been associated with the computer, 
students nuy have expected to use the system in the test particular sessions. In ordo- to 
prevent expwtancy effects from distorting the pre^:lass daily rcpotu the computer was present 
m the room dunng the mtroducdons for all sections and in plain v!«nv of all students To 
further minimize expectancy effects among the students, during die year the experimenter also 
admimstered Ac questionnaues in class meetings in which there were no experimental 
con^ansons bemg conducted. Once aU students were finished wi A Ae prc-class report, Ae 
expCTimcnter assigned students to their lab group, and then proce^ to record obswvations of 
teacher and student behaviors. In those labs in which boA instructional media were present, 
Ae experimenter alternated recording observations for minute-long periods between Ae 
compute group and Ae control group. When Ae first lab group finished Ae procedures 
observations were terminated and Ae post-class questionnaires were handed out tn each erouD 
as mey fimshed. ^ *^ 

Design and Data An;:lyses 

^ ^ ^ * evaluation was structured as a series of 19 independent experiments, each wiA two 
treatments: Ae experimental (EDAS computer-based) and the control (traditional apparatos) 
Vanablw were clustered in three predictor domains and one criterion domain as indcated in 
Figure in.l. 

AH _ ^'L**?*"^t was first analyzed separately wiA a complete multivariate analyses. 
AU Pferson, Environment, and Outcome variables were tested wiA a multivariate analysis of 
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variance Hien each member of the domain was examined with univariate analyses of 
covariancc. using a general linear model. Factor analyses and regression analyses during the 
pilot iMiase of the evaluation indicated that a student's grade in the science course being studied 
was the most appmpnatt indicator of ability, Aus it was used as the covariate. Eariy in the 
year recent grades in the science course served as covariate. Once a quarter course grade was 
available it served as covariate. When semester grades became available, they were used as the 
covanate. When the analysis of covariance showed no significance for the covariate, univariate 
analyses of variance were employed. 
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[Hd Study habits 
[iviti^nd attitudes 

lies 



Person 

learning 
school a( 
personam 
demogr£ 

attitudes tcLvards class ^ 
attitudes towards computei^^ 
attitudes towards laboratory 

courses 
grades 



Behavior — 

off task 

invovlvement with group 
individual attention fro^^^cher 
group attention toja^er 
attention to ^p|9^tus 
writing 



Outcome 

Lab reports 

Exams 
facts 
principles 
applications 



Environment 

science discipline 
mode of use 
time of year 

degree of experience with computers 

F i gure ni . l . The domains of the variables and the sets of variables that were examined during 
me experiments for their impact on the educational outcome. 
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TTie behaviocal data were obtt ned as frequency of occurrence for each of the six 
categoncs of student behavior and three categ<Kies of instructor behavior. These daia were 
analyzed with chi-squarc. Wher- there were cells that had too few observations to meet 
stringent chi-square criteria, fee ^ucncies were colk jsed across cells. 

After all 19 experiments nad been analyzed independentfy with the multivariate 
procodmcs. mct^Mnalysis techniques were employed to examine the data from aU experiments 
amultaiwously . For aU variables but the behavioral, effect sizes were computed diiwdy from 
Je raw date m each experiment For die behavioral variables, the effect sizes were coirouted 
from the chi-square results. To detea possible inlenurtions between treatments under different 
conditions, the meta-analyses enq>loycd coding, or indicator, variables. The major 
cnvffonmentel vffliables that served as indicators were: (a) mode of use of the apparatus, 
whether actively by the students or in an im luctor led demonstrations, (b) demof 

ejqjcnencc with conqwteis, which varied from courses where Ac EDAS was the only 
compatam use to courses where a variety of conqniters were used for a varied of dtfferent 
purposes by botii st^ents and instructor, and (c) time of year, varying between the stert of the 
year wh«novelQr effects couM stin be occurring and the midd^ 
sch<»l woA was ma:mud for both instructor and student and aU pi^^ 
tamiliar. Fwison product momert correlation coefficients were obtained between each 
mduatormdtheeffectsizesforcachvariable. Where interactions between indicators were 
possible, the correlations were obtained by examimn«? each ^evel of each indicator, in tura and 
conqwtmg a conelation matrix between the other incucators and the variable effect sizes. ' 

RESULTS 

1. computer-based treatment differed from the traditional in its effect on variables in 

all (rfthe domains. The metaranalyses indicated Aat these effects showed regularity across the 
oitterent expenments. Each <rf the indicator variables was associated with significant 
differences in die effect of Ae two treatments. 

BoA the huge amount of date to be summarized and Ae coherence of the picture that 
ema^ from the analyses make a tabular presentation of the results most effective. 
Accordmgly, Tables IIL2 through IIL5 summarize the findings. WiA effect size computetions 
m mcte-aralys«, rqpatmg Ae actual means averaged across experiments can be mislradrng 
andmvahd, as the difference between averages of means is not directly proportion 
effect size and may not even have the same sign. Therefore, only representative means and 
variance analyses will be repotted in Table IIL5. Tables in.2 thrtnigh in.4 present the mcte- 
analysis findings, co^unmg boA Ae coiretetion coefficients d) between Ae indicators and Ae 
effect sizes and the effect sizes Oi). themselves, for aU significant mete-analytic results. 

• ji?*^ ^ mete-analysis results are pi^ented for each of the indicators 
considered separate^. For Ae indicator, time of year, Ae computer group changed from 
r^ortmg more comfort wi A computers to repotting less comfort than Ae traditional group 
With cnjoym^of computus the computer group rq)orted less enjoyment Aan the traditional 
atnudyear. Off task bclavior for the computer group changed from more Aan that for Ae 
traditional at the start fo Ae year to less Aan the traditional by mid year. Attention to Ae 
g)panitusshowed an opposite effect Amount of writing became lower for Ae computer group 
^rff^^i^ ^ T^y^- H^iPPOrt scores, which were higher for Ae compiler gSups 
at the start of the year, showed no difference across groups at midyear f f» 
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For the uKhcator.nxxic of use, the computer Students m the active use grou^ 
IMS interest in the course, a less difficult class, and a less rushed class than did the traditional, 
wntmg behavior was less for the computer students than for the traditional in the active use 
classes., but exam ^licatim scores were higher. 

For tiie degree of experience indicator, when experience was high ihc computer users 
reported less confidence in their abiUties than did the traditional The behavior of Sie computer 
groups showed mere off task activities than that for the traditional Exam performance on both 
pnnaplc and ^bcanon questions, as weU as on tix sum of aU laboratory-relevant questions 
was higher for die computer groups than for tiie traditional 
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Table IIL2 

Global Meta- Analysis Re5sults For Each Tndicatnr 



Time of Year (1 = start, 2 = middle) 



D 


Variable 






12 






d2 




P 


Comfort with computers 


.48 


<.05 


-.76 


<.001 


.28 


-.34 


19 


P 


Enjoyment of conq)uters 






-.69 


<.001 


.30 


-.42 


19 


B 


Off task behavior 


.56 


.01 


-.57 


.01 


.33 


-.55 


19 


B 


A J^tion to ^jparatus 


-.53 


<.05 


.56 


.01 


-.28 


.45 


19 


B 


Wriviig 






-.56 


.01 


-.14 


-.92 


19 


O 


Lab report: total score 


.85 


<.10 






1.04 


.00 


19 



Mode of Use (1 = student, 2 = demonstration) 
Variable ri2 B di d2 li 



p 


Interest in course 


-.44 


<.06 


-.21 


.07 


19 


p 


Difficalty of class 


63 


<.01 


-.47 


.09 


19 


p 


Rushed class 


-.61 


<.01 


-.44 


.37 


19 


B 


Writing 


-.65 


<.01 


-.88 


-.03 


19 


O 


Exam: applications 


.67 


.001 


.39 


-.22 


19 
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Table III.2 (continued) 



Degree of Experience (1 = inexperienced, 2 = experienced) 



D 


Variable 


112 


0 


dl 


d2 




P 


Confidence in ability 


-.47 


<.05 


.17 


-.25 


19 


B 


Off task behavior 


.46 


<.05 


-.27 


.25 


19 


0 


Exam: lab questions 


.60 


<.01 


-.04 


.28 


19 


0 


Exam principle questions 


.50 


<.10 


-.02 


.32 


19 


0 


Exam: applicatim questions 


.68 


.001 


-.22 


.39 


19 



H^. D IS the domain ftom which the variables are sampled: where P is the person. B is the 
behavior, E is the ejivironment, and O is the outcome domai-.. The number of experiments that 
dl and jfe arc each based upon is indicated by H. When a correlation is not significant, it is 
omitted ftom the table and the comesponding effect size is obtained fiom the correlation for the 
opposite level of the indicator. 
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The effects of the mode of use also differed across levels of the other indicator 
variables, as presented in Table in.3. Consider the mode of active student use first. When the 
laboratory apparatus was employed by small lab groups rather than by the whole lab section, 
computer groups reported more motivation for class and experienced less time receivring 
mdividual attention finom the teacher than did traditional sections. The time of year indicator 
revealed similar effects as iv did across all modes of instruction with the exception that group 
attention from the teacher was less by midyear for the computer groups than for the traditional 
Companng courses with high conq)uter experience to those with little computer expmence 
revMted student reports for the experienced computer users of lower interest in clai«, less 
confidence m their ability, and less sense of knowing what they were doing, and obi?erver 
reports or more group attention from the teacher than was found with the traditional aooantus 
usears 
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Table m.3 

Meta- Analysis Results F or Indicafnit Mode of CompiitCT TTsp 



Student active use 



mdicator 


D 


Variable 


£ 




dl 


d2 




Size of lab group 


P 


Motivation for class 


.75 


<.05 


-.27 


.7-3 


8 




P 


Individual attention 


-.71 


<.05 


.47 


-.17 


8 


Time of year 




see Table 2 also 














B 


Attenti<»i to teacher 


-.68 


<.10 


.29 


-.23 


8 


Degree of experience 


P 


Interest in class 


-.71 


>.05 


.19 


-.53 


8 




P 


Confidence in ability 


-.69 


<.10 


.41 


-.34 


8 




P 


Know what doing 


-.75 


<.05 


.37 


-.34 


8 




B 


Attention to teacher 


.68 


<.iC 


-.23 


.29 


8 






Instructor led demonstration 








Indicator 


D 


Variable 


I 


12 


dl 


d2 




Time of year 




pee Table 2 also 














P 


Confidence in ability 


-.74 


<.01 


.40 


-.10 


11 




P 


Know what doing 


-.62 


<.05 


.32 


-.18 


11 




B 


Attenticm to teacher 


.54 


<.10 


-.41 


.08 


11 




B 


Writing 


not significant 










0 


Exam: applicati(»is 


.57 


<.10 


-.31 


-.05 


11 




0 


Exam: total 


«^ 


<.10 


-.04 


.40 


11 
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Table 10.3 (continued) 



Instructor led demonstration 

indicator 



D 


Variable 


I 




dl 


d2 


P 


Motivation for course 


.53 


<.10 


-.02 


.54 


P 


Interest in class 


.55 


<.10 


-.15 


.33 


p 


Leanml in Hacc 

1 <Vimi^A* 111 WlOdO 


.Ol 


^ 1 n 
<. lU 


-.Ul 


.53 


P 


Difficulty of class 


.60 


<.10 


-.20 


.57 


B 


Off task behavior 


.77 


<.01 


-.20 


.71 


B 


Group involvement 


.64 


<.05 


-.05 


.23 


B 


Attenticm to teacher 


-.79 


<.01 


.10 


-.73 


B 


Attention to apparatus 


-.78 


<.01 


.10 


-.73 


0 


Exam: lab questions 


.67 


<.05 


-.07 


.34 


0 


Exam: principles 


.63 


<.05 


-.01 


.34 



Mate. The mdicator is the coding characteristic of the laboratory class meeting that is 
employed m the analysis to create two categories for comparison. These indicator paired- 
comparisons are (a) size of lab group: whole class vs small groups, (b) time of year: start vs 
nuddle, and (c) degree of experience with computers: Uttle vs much. D is the domain from 
which the variables are sampled: P = person, B = behavior, E = environment, and O = 
outcome. The effect size for the first named member of the paired comparisons is di and for 
the second named member of the pair, ^. The number of experiments that di is based upon is 
indicated by U. The sum of the two N's for ui and d2 is 19. For time of year, the statistics 
arc whenever available, averages of those for the separate analyses of start vs rest of year and 
middle versus rest of yeax. 
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Consider next the effects obtained when teachers led demonstrations with the laboratory 
apparatus. The effects of time of year were similar to those across all modes of use with a few 
additions. By midyear Ae increased confidence and knowing what they were doing, relative to 
the fraditional groups, was lost for the students in the computer groups. The computer groups 
received mate group attention ftom the teacher than the traditional by midyear. Finally, the 
application questions on the exams became equal across conditions by midyear, while tiie total 
exMi score for the ccnputer groups rose above that for the traditional at midyear. In courees 
with high computer experience demonstrations resulted in the following changes for the 
conyuter groups when compared to the traditional. The computer groups showed more 
motivatwn, more interest in class, and reported that they learned more, though the class was 
more difficult than for the traditional groups. Off ta&V and group involvement behaviors 
increased for fbc computer groups when conqjared to the traditional, while attention to the 
teacher and Hhc q>paratus decreased relative to the traditional. Exam scores on both the 
imnaple questions and the sum of all questions directiy relevant to the laboratoiy exercises 
TOiC for the computer groups conq>ared to Aose for the traditional 

The in^t of the degree of experience with computers for teacher and students varied 
^TOSs levels of ibe other indicator variables, as well, and is presented in Table in.4. Consider 
the courses with experienced users first The time of year effect was similar to that previously 
reported with a few additions. By mid year the difficulty of the course increased and the 
indepoQdencc of the laboratoiy activities decreased for the computer groups when compared to 
Uie ttaditional. On the exams both the principle and the application question scores were hiisher 
for the computer than for the traditional at midyear. 
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Table IIL4 

Meta-Analvsis Results For Tnriiratnr TVg»> ^ of Computer Experiennft 

Experienced with computers 
Indicator D Variable IB ^ H 



Time of year see Table 2 also 

P Difficulty of course .75 <.10 .00 .84 6 

P Independence in lab -.91 <.01 -.05 -2.18 6 

B Off task behavior not significant 

B Attention to apparatus not significant 

O Exam: principles .82 <.05 .19 1.00 6 

O Exam: plications .98 <.001 .03 1.00 6 
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Table IIL4 (continued) 



Inexperienced with computers 



Indicator 



Time of year 



D 



Variable 



see Table 2 also 

P Enjoyment of course 

P Lear ^ in class 

B Off task behavior 

O Lab t sport total 

O Ex^m; total 

O Exam: applications 



not significant 
.53 <.10 -.55 

not significant 

not significant 
.61 <.05 -.04 
.59 <.05 -.31 



-.04 



13 



.38 
-.04 



13 
13 



Mfltfi. The indicate/ is the coding characteristic of the laboratory class meeting that is 
employed in the analysis to create two categ<wies for comparison. For time of year the two 
categories are start vs middle of year. D is the domain from which the variables are sampled: P 
= persai, B = behavior, E = environment, and O =: outcome. Thr effect size for the start of the 
year is ds and for the middle, jJm- The number of observations mat ds is based upon is 
indicated by Ji- The sum of the two N's for and is 19. The statistics are, whenever 
available, averages of those for the separate analyses of start vs rest of year and middle versus 
rest of year. 
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Eariie- results can be restated for experienced users with demonstrations as follows 
The conqniter users, when compared with tpwiitional apparatus usere, reported for 
dcmonstr^ons more interest in the course and in class meetings, more sense of learning, but 
©tater difficulty, and shovyeC less attention to -le teacher than they reported t^d showed when 
tbcy were in active use of the apparatus. 

Next consider the users with less experience. Time of year effects were, again, similar 
to the ovcraL results of time of year with a few exceptions. The differences in enjoyment of 
conq)utcrs, off task behavior, and U* report scores were not present Changes not revealed in 
the ovcraU analysis included for the computer groups less sense of learning in class at the start 
of the year < was felt by die traditional groups. In addition, the computer groups performed 
woisc on the total exam at die beginning of die year but better by mi<fyear thandid thSr 
toiditiond counterparts. A tow exam appUcation score for the computer groups at uie start of 
the year disappeared by midyear when both computer and traditional groups were equal on this 
measure. ^ ui« 

J ^ berestated for inexperienced users with demonstrations as foUows. 

""l^r*? '^^^ ^f^* computer groups reported less interest, 
confidence m their ability, fedmgs that the methods used were not effective; and sense of 

reporting more diTiculty and sense of being rushed Uian 
the traditional group members reported. Group involvement and attentioi. xo the apparatus 
were also less, while attentic. to the teacher and writing were greater for these computer users 
than for the traditional apparatus users. 
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Table IIL5 

Representative Means anH Analysis Results 



Person domain 



Variable 


1 imp 

Period 


Means 

Computer Tn^ditional 


E 




it 


Difficiiltv of HaQQ 


pre + puai 


2.60 


2.10 


3.98 


1,28 


.05 


Eniovment f\f 
computers 


pre A pOSl 


2.34 
3.16 


3.29 
3.21 


12.13 


1,42 


.00] 


i^.oniroF witn 
computers 


pre + post 


4.08 


4.42 


6.41 


1,57 


.01 


Confidence in 
ability 


post 


3.65 


5.35 


5.74 


1,22 


.02 


Know what doing 


post 


3.36 


4.10 


6.32 


1,32 


.02 


Motivated 


pre 


3.33 


2.60 


5.96 


1,28 


.02 


Interest in 
class 


pre + post 


2.43 
3.00 


2.59 
2.68 


6.16 


1,34 


.02 


Amount learned 


post 


3.53 


3.15 


4.65 


1,62 


.02 


Rushed 


post 


2.15 


1.59 


4.58 


1,58 


.03 


Effectiveness of 
methods 


post 


2.79 


3.50 


4.03 


1,32 


.05 


Independence 


post 


3.53 


4.14 


4.05 


1,22 


.05 
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Table ELS (continued) 



Behavior domain 



Variable 


Sign of 


Means 
Computer Traditional 


X2 


df 




Off task 


+ 


.292 


.116 


59.74 


1 


<.0001 






.119 


.196 


13.84 


1 


.0003 


Group involvement 


+ 


.72 


.65 


8.56 


2 


.01 






,546 


.597 


4.14 


1 


.04 


Individual attention 




.120 


.161 


8.39 


2 


.02 


Attention to teacher 


+ 


.222 


.139 


26.44 


2 


<.0001 






.155 


.216 


9.67 


2 


.008 


Attention to ^paratus 


+ 


.960 


.897 


13.01 


2 


.00? 






.642 


.748 


14.04 


1 


.0003 


Writiir 




.126 


.064 


12.14 


2 


.00^ 






.253 


.352 


8.57 


1 


.004 
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Table IIL5 (continued) 



Outcome domain 



Variable 


Tmie 


Means 


E 








Period 


Computer 


Traditional 






Lab lepoft tots! 


post 


10.00 


7.17 


12.98 


1,30 


.001 


Exam: total 


post 


10.00 


8.77 


4.06 


1,57 


.05 


Exam: lab questions 


post 


10.00 


7.59 


4.85 


1,57 


.03 


Exam: principles 


post 












Exam: applications 


post 


10.00 


3.11 


6.55 


1,57 


.01 



KfilC. For each row of the table with the person and outcome variables an analvsis of 
aSIi^""-!? T^"* that represents the difference score results of the eariier met 

aralyses Tie tunc period indicates whether the variable was measured just before (pre) or at 
S^IS?"*^ ^^L?^^'- ^* P^«* « presented, or/.y that period was si^cant 

When bom penods ac summed, only the main effect of treatment was significant When the 
Z^J^?? multiplied, the icteraction over time was significant and is presented in the table 
next to the post scores. AU attitude scales range ftom a low score of 1 to a high of 5. Outeome 
measures have been standaidized so that the experimental group score is set at 10 For the 
behavior variables a chi-square aialysis is reported that represents the difference score results 

Behavioral scales arc proportions of time in which the class was 
?^ engaged ui that behavior. The sign of d column reports whether the effect size 
indicated difference mfevw -y^^ 
they existed, results lor both directions of effect sizes are reported. 
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DISCUSSION 



A complex though comprehensible picture emerged of the multidimensional and 
multidomam nature of «iucational outcome fiom CBI in science teaching. TTie conflicting 
f ^fflli^Sfr^V'^t.*"'^?* ^SS^ & (1980) and the small aSitude 

ffSSnT^ (1^^^^ ^ H°^tandable in light of the present results. Most 

mterestmg were the differences between attitudes and performance. In some cases, such as for 
courseswith a high degree of computer experience where the demonstration mode was used 
unproved attitudes were dnectly proportiooal to increased achievement Yet when 
d<monstr^ions were con^ared with active student use of the computer, the demoastration 
moderwulledmm^wovedattmidcsbutd^creastt^ Our knowledge has advanced 

to Ae stage where research on CBI mur; se.>n;h for the confounding variables and seek to 
undtetand the comrlexity of the undc.lying process rather than sinmly demonstrate a 
diiTcxcncc m scores. 

, . the design of the study wc^fou^^ 

r^mdctcrmmmgAeminictofth^ Time of year, degrei of computer exVeri^S 
mock of use each produce mam effects on outcome and interact with 
significfflitwsqrs. No smgle condition is optimaL Instructors must choose tools and 
approaches * ^ Soals; they cannot, unfortunately, simple choose the 
umversaUybesttreatmeit. 

One rathCT genaal effect was the change in attitudes towards computsi?, the course 
andAemselves that students reported with increased exposure to computers. When computer 
ocposure moeascd, whether through passage of time during the academic year, by being in a 
class with extensive con^uter use, or by using the computer actively rather than observing a 
^racmstrjion,studratsre^KwtedlessIiki^^ Often under these 

sajic COTditions students also reported less confidence in then- own abilities and a diirSnished 

^^^ J^S"^!^^*^ ^'^^ ^ *^ ™c*ais ^^"i used in class were 

effective. It is m^ratant to rerocnu>r' that these decreases were not fiom positive to negative 
fcelm©5, but were finm very positive to somewhat less positive (fiom 4.4 to 4.0 on a 5 point 
scale, for example). Field observations suggested two related reasons for this change. With 
mrae co^jto e3?«oftK» sti^ 

hSSJ? if* thought that the introduction of the con^puter would 

^^'^^X^^^^yJ^^ m lab. As the year progressed they discovered tf^at. whfle the 
^^^r^^y^^^f'^^^'^^'^pl^^ Firthermore,b<3rstudents 
!f?c?^^Tl ?^ ^^^^ conqwter did not make learning sin^ler in a c.>gnitive 

^f»^bothfoundth7:itecampnterinudelearnm 
fZ^i^S^V"^^ ofAc 3 dting and othtr cleric^ taste, the conq-uter fiSd suuS^a^^* 
teachtt to flunk mtxe abou tl«e content of what they were doing. The ease and speed of 
coUectag<UiaTneamdwi Acre was rnore^ Soft the 

teachers and stidents found this ina«ase in cognitive responsibility to be harder and more 
f^^Ils^i *f ^-P^*:' Smostrote acdvitie? of earUer lab experiences studei.ts 
found saence to &c naid wwk, especially when the new tool clearly removed the rote activities 
This suggeste th^ Jie cctiq>uter accelerated the contrast between simple and more professional " 
ST^W^h^"" ^ ^""^^^ ^^^'^^ ^ "^""^^ between junior high 
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CoocuiiOTt With this decrease in attitudes was an increase 
S^^S i'^^ engaged in less time off task during the laboratoiy and shifted tlieir 
iJ^^S^SPi?*'^ ^ ^ attending to the apparatus. This improvement in tehavior was 
n^.t dirccay hnked to an improvement m their laboratoiy reports; the computer groups initiaUv 
wrote better than the ofho^, but this difference di^«akrf by SS^ftte yS? 

computer experience there also emaSSThniJovement in exSSi 
E^SSf S ^ improvement did not occur as clearly for the classes with 

less ft»cher and student experience. Future work must carefully distinguish between 
expenencca fuid inexperienced classes when examining CBL 

The mode in wl^h the computer was use.' significantly influenced the imiw^ 
conjgi^wistodents. One of tte contrasts between the modcJ of use was that be^«^^ 

S?S^^"'S^f"^*'^' results indicate that in the inexperienced classes 
the demonstration mode was kj» successful than the active student use mode. Field 

*is effect iras due to the experience of the teacher more than that 
2^ St J!??S!22??SS J? documented by previous work (StimmJ 

QMmor, .,4cCaskiM, & Dunrett. 1981). The behavior of the teacher c£ also influent the level 

* cxperknci are most ex^ed lo 

Aestudcnts. Inexpasca^dtMchers were less confidem»ndinore 

^Jt"^.^^"^!^^- "n^attitwiesaiKibehavioisofSeS^ 

reflect these differences, though the students' performance on examinations was no^ iffected. 

A^r^^t^^t^^ ^^"^ ^""P^ a great oeai fitom their computer 

behavioral and ou^ome data do 
" amtudes, espcdaUy for classes with computer expuiencTfo 
fS^f ^^r»f^ ^ appear at first to be confusing for these classes . While m^atioo and 
r terest was higher, there was more off task behavior and less attention to the teSS die 
aijaratos. This an^y can be understood by examining the group involvement behavior 

observation reve^ tins inctSTa^SS 

STaSS^h While die discussion 

was assocwted wifli less traditionaUy expected measures m tajik behavior, it. nevertheless 
was related to mqm>ved abiUly to remei^ die relev.mt pmiciples ^^y tiwm SSw 

c- A« J^^u^^^ laboratory exercises are traditionally used botfi to familiarize 
students v/ith Uie metirods and iq>paratus of scic.icc and to provide immediate examples of 
physical principles and relationships. In the first of these purposes the science lab exercise 
nuyvery wcUsua«edtothccxtentthatprccisioni^ hits 
second pu^we, the soexd lab may not be serving directiy and immediately to add much to the 
JS^^ Iff ^ concepts Data taken during ck s regains in tabular firm ts 
TJ^^J^.^^ f ^u* ^^"^^ exei. JSC. While tiiosc witii steacfy study habits 
may be prompt m wm^Jcting such exercises, for most the principle or relationship which tiie 

SfdT^S?^"^ ^ ^ Sass tsriod It is only whc^ 

trSSS.}^ ? * specificaUy meamngM form tiiat they begin to be oi^anized in easily 
rcc^le fashion. It therefore stands to reason i.^t the more tiie data assume tfie nec^saiy 

S^ri?. ^^I^Z w P^'P? at die time of collection, the longer die students have to 
practice wiQi this fonn of the data. 
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V >ne mightexpect, then, that the active student labs would be more effective than the 
dOTionstnmons. The actual picture is not quite that simple. Student labs were seen as less 
mteresung than demonstrations, but were also reported to easier and less rushed Writing 
behavior was also Iws for student labs. Apparently the use of the comnuter did eliminate some 
of the clcncal work load. Key to understanding the student tos ar*. he jxamination data. 
Apphcati<m abihty unproved for students when they actively used the conq>uter in class. There 
K some indicabon that the leammg strategies differed in the active student labs (see Weinstein 
«Mayer, 1986, on strategies), and this resulted m an increased abiUty to use more broadly 
what w& Iwmed. It is important to note that active student use of the conmuter was not 
gcnoaUy effsctive for aU types of examination performance; its effect was focused on the 
ability to ap{dy. 

The results of the 19 experiments do have one common pattern to them. Technology 
mprowde took for ksmmg, b»rt it is what people do with the tools that determines their 
effect. Asar^edby Whiteandrisher(1986).cuiriculumisamatterofpeople. Thepresent 
data have confirmed that what compu* . s do is provide a potential technique for helping 

if '''^^P, ^f^^ reasoning slQlls and an enhanced understanding of science (Arons, 
1984). Ine results turther suggest how this potential is converted into rwlity 
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